|
2014-04-08 00:00:35
|
<RoanKattouw>
|
Sure thing
|
|
2014-04-08 00:00:42
|
<RoanKattouw>
|
I think that's the SWAT all done
|
|
2014-04-08 00:00:44
|
<RoanKattouw>
|
Sorry for the slowness everyone
|
|
2014-04-08 00:01:16
|
<bd808>
|
RoanKattouw: If it makes my mailbox less full of debate about font faces...
|
|
2014-04-08 00:01:36
|
<bd808>
|
is sure that muting those threads will continue
|
|
2014-04-08 00:02:28
|
<icinga-wm>
|
PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 7.426 second response time
|
|
2014-04-08 00:08:52
|
<bd808>
|
looks for a python reviewer for: https://gerrit.wikimedia.org/r/#/c/124500/
|
|
2014-04-08 00:09:10
|
<bd808>
|
I think that will fix the 1.23wmf21 l10n problems
|
|
2014-04-08 00:09:30
|
<bd808>
|
Because … mystery action at a distance!
|
|
2014-04-08 00:12:27
|
<icinga-wm>
|
RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 219118 bytes in 8.455 second response time
|
|
2014-04-08 00:24:56
|
<logmsgbot>
|
!log catrope synchronized php-1.23wmf20/extensions/VisualEditor 'it helps if you run git submodule update first'
|
|
2014-04-08 00:25:02
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 00:25:05
|
<logmsgbot>
|
!log catrope synchronized php-1.23wmf21/extensions/VisualEditor 'it helps if you run git submodule update first'
|
|
2014-04-08 00:25:11
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 00:27:34
|
<grrrit-wm>
|
('PS1') 'BryanDavis': test2wiki to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124505'
|
|
2014-04-08 00:28:54
|
<bd808>
|
RoanKattouw_away: Are you {{done}} done now? I'd like to run some more scap tests
|
|
2014-04-08 00:38:27
|
<grrrit-wm>
|
('Abandoned') 'BryanDavis': l10nupdate: Add temporary debugging captures [operations/puppet] - 'https://gerrit.wikimedia.org/r/124467' (owner: 'BryanDavis')
|
|
2014-04-08 00:38:40
|
<grrrit-wm>
|
('PS2') 'BryanDavis': test2wiki to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124505'
|
|
2014-04-08 00:39:44
|
<grrrit-wm>
|
('Abandoned') 'BryanDavis': test2wiki to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124505' (owner: 'BryanDavis')
|
|
2014-04-08 00:41:34
|
<grrrit-wm>
|
('PS1') 'BryanDavis': Group0 wikis to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124506'
|
|
2014-04-08 00:43:55
|
<bd808>
|
greg-g: Are you still on a bus? I'd like to scap group0 to 1.23wmf21 to test my band aid fix. I would be on the hook to revert immediately following if ExtensionMessages looks like it will cause a problem for l10nupdate.
|
|
2014-04-08 00:44:03
|
<RoanKattouw_away>
|
bd808: Yes, sorry
|
|
2014-04-08 00:44:43
|
<bd808>
|
RoanKattouw_away: :) thanks. I watched your idle time on tin climb until I felt safe.
|
|
2014-04-08 00:45:28
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 00:46:57
|
<bd808>
|
decides that greg-g won't have changed his mind in the last 1:30 and proceeds
|
|
2014-04-08 00:48:38
|
<grrrit-wm>
|
('CR') 'BryanDavis': [C: '2'] "Approving to test band aid fix for ExtensionMessages generation problem. Will revert if ExtensionMessages doesn't look right after scap." [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124506' (owner: 'BryanDavis')
|
|
2014-04-08 00:48:45
|
<grrrit-wm>
|
('Merged') 'jenkins-bot': Group0 wikis to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124506' (owner: 'BryanDavis')
|
|
2014-04-08 00:50:53
|
<logmsgbot>
|
!log bd808 Started scap: group0 to 1.23wmf21 (testing python change for mwversionsinuse)
|
|
2014-04-08 00:50:58
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 00:53:12
|
<bd808>
|
sees l10n cache updating yet again for 1.23wmf21 and loses all confidence in his "fix"
|
|
2014-04-08 00:53:51
|
<logmsgbot>
|
!log bd808 scap aborted: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (duration: 02m 57s)
|
|
2014-04-08 00:53:56
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 00:54:30
|
<logmsgbot>
|
!log bd808 Started scap: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (again)
|
|
2014-04-08 00:54:35
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 00:54:56
|
<logmsgbot>
|
!log bd808 scap aborted: group0 to 1.23wmf21 (testing python change for mwversionsinuse) (again) (duration: 00m 25s)
|
|
2014-04-08 00:55:01
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 00:55:12
|
<grrrit-wm>
|
('PS1') 'BryanDavis': Revert "Group0 wikis to 1.23wmf21" [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124507'
|
|
2014-04-08 00:55:34
|
<grrrit-wm>
|
('CR') 'BryanDavis': [C: '2'] Revert "Group0 wikis to 1.23wmf21" [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124507' (owner: 'BryanDavis')
|
|
2014-04-08 00:55:42
|
<grrrit-wm>
|
('Merged') 'jenkins-bot': Revert "Group0 wikis to 1.23wmf21" [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124507' (owner: 'BryanDavis')
|
|
2014-04-08 00:56:51
|
<logmsgbot>
|
!log bd808 Started scap: revert group0 to 1.23wmf21 (testwiki still on 1.23wmf21)
|
|
2014-04-08 00:56:55
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 01:01:33
|
<grrrit-wm>
|
('PS3') 'Ori.livneh': Add EventLogging Kafka writer plug-in [operations/puppet] - 'https://gerrit.wikimedia.org/r/85337'
|
|
2014-04-08 01:06:45
|
<logmsgbot>
|
!log bd808 Finished scap: revert group0 to 1.23wmf21 (testwiki still on 1.23wmf21) (duration: 09m 54s)
|
|
2014-04-08 01:06:53
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 01:22:25
|
<StevenW>
|
ori: working now
|
|
2014-04-08 01:22:29
|
<StevenW>
|
\o/
|
|
2014-04-08 02:07:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 02:07:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 02:07:08
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 02:07:08
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 02:15:58
|
<logmsgbot>
|
!log LocalisationUpdate completed (1.23wmf20) at 2014-04-08 02:15:58+00:00
|
|
2014-04-08 02:16:06
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 02:34:57
|
<logmsgbot>
|
!log LocalisationUpdate completed (1.23wmf21) at 2014-04-08 02:34:56+00:00
|
|
2014-04-08 02:35:02
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 02:45:57
|
<icinga-wm>
|
PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
|
|
2014-04-08 02:48:37
|
<icinga-wm>
|
PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
|
|
2014-04-08 02:48:57
|
<icinga-wm>
|
RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds
|
|
2014-04-08 02:49:06
|
<ori>
|
springle_: db1047 has been very sad lately
|
|
2014-04-08 02:49:27
|
<icinga-wm>
|
RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds
|
|
2014-04-08 03:00:17
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 03:08:06
|
<bawolff>
|
With 1.23wmf21 not getting deployed to mediawiki.org last thursday, does that mean the deployment schedule for 1.23wmf22 will be off by a week?
|
|
2014-04-08 03:11:07
|
<logmsgbot>
|
!log LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 8 03:11:04 UTC 2014 (duration 11m 3s)
|
|
2014-04-08 03:11:11
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 03:31:47
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 03:38:12
|
<aude>
|
greg-g: still around?
|
|
2014-04-08 03:53:36
|
<aude>
|
greg-g: check your mail
|
|
2014-04-08 04:03:35
|
<TimStarling>
|
!log upgrading libssl on ssl1001,ssl1002,ssl1003,ssl1004,ssl1005,ssl1006,ssl1007,ssl1008,ssl1009,ssl3001.esams.wikimedia.org,ssl3002.esams.wikimedia.org,ssl3003.esams.wikimedia.org
|
|
2014-04-08 04:03:41
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 04:03:57
|
<Jasper_Deng>
|
TimStarling: is this the heartbleed.com thing?
|
|
2014-04-08 04:04:07
|
<Jasper_Deng>
|
didn't know we used openssl
|
|
2014-04-08 04:15:22
|
<TimStarling>
|
Jasper_Deng: yes
|
|
2014-04-08 04:15:47
|
<TimStarling>
|
!log also upgraded libssl on cp4001-4019. Restarted nginx on these servers and also the previous list.
|
|
2014-04-08 04:15:51
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 04:37:40
|
<Ryan_Lane>
|
!log upgrading libssl on virt1000
|
|
2014-04-08 04:37:44
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 04:38:21
|
<Ryan_Lane>
|
!log upgrading libssl on virt0
|
|
2014-04-08 04:38:26
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 04:41:03
|
<TimStarling>
|
!log upgraded libssl on zirconium.wikimedia.org,neon.wikimedia.org,netmon1001.wikimedia.org,iodine.wikimedia.org,ytterbium.wikimedia.org,gerrit.wikimedia.org,virt1000.wikimedia.org,labs-ns1.wikimedia.org,stat1001.wikimedia.org
|
|
2014-04-08 04:43:13
|
<TimStarling>
|
!log restarted apache on the above list, failed on labs-ns1, virt1000, ytterbium
|
|
2014-04-08 04:43:18
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 04:43:47
|
<^d>
|
TimStarling: I'll poke ytterbium
|
|
2014-04-08 04:44:00
|
<^d>
|
Keep moving on to other boxes if you need.
|
|
2014-04-08 04:44:35
|
<^d>
|
Seems up now.
|
|
2014-04-08 04:45:04
|
<TimStarling>
|
yeah, labs-ns1 and virt1000 are actually the same server
|
|
2014-04-08 04:45:19
|
<TimStarling>
|
and apache is running there with stime after the upgrade
|
|
2014-04-08 04:46:30
|
<TimStarling>
|
!log on dataset1001: upgraded libssl and restarted lighttpd
|
|
2014-04-08 04:46:34
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 04:53:47
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 05:08:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 05:08:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 05:08:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 05:08:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 05:25:10
|
<grrrit-wm>
|
('PS1') 'Aude': Enable Wikibase on Wikiquote [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124516'
|
|
2014-04-08 05:26:24
|
<grrrit-wm>
|
('CR') 'Aude': [C: '-2'] "requires sites and site_identifiers tables to be added and populated on wikiquote" [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124516' (owner: 'Aude')
|
|
2014-04-08 05:31:00
|
<_joe_>
|
!log upgraded openssl on cp10* and cp30* servers as well
|
|
2014-04-08 05:31:06
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 05:39:29
|
<apergos>
|
!log restarted apache on fenari magnesium yterrbium antimony
|
|
2014-04-08 05:39:33
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 05:39:51
|
<apergos>
|
with some mispellings but people will get the point
|
|
2014-04-08 05:47:01
|
<apergos>
|
!log shot many old apache processes running as stats user from 2013, on stat1001 (restarting apache runs it as www-data user)
|
|
2014-04-08 05:47:06
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 06:34:37
|
<grrrit-wm>
|
('PS3') 'Matanya': dataset: fix module path [operations/puppet] - 'https://gerrit.wikimedia.org/r/119212'
|
|
2014-04-08 06:37:44
|
<grrrit-wm>
|
('PS3') 'Matanya': exim: fix scoping [operations/puppet] - 'https://gerrit.wikimedia.org/r/119496'
|
|
2014-04-08 06:43:48
|
<matanya>
|
springle: did you hear from otto regarding https://gerrit.wikimedia.org/r/#/c/122406/ ?
|
|
2014-04-08 06:45:27
|
<springle>
|
matanya: no
|
|
2014-04-08 06:45:41
|
<matanya>
|
:/ i need to chase him down, thanks
|
|
2014-04-08 06:46:04
|
<springle>
|
not sure otto knows about it? i emailed analytics lists directly
|
|
2014-04-08 06:46:29
|
<springle>
|
so far the answer is: probably fine to decom db67, but lets wait for enveryone to chime in
|
|
2014-04-08 06:46:43
|
<springle>
|
i'll bump it this week
|
|
2014-04-08 06:47:05
|
<matanya>
|
thank you
|
|
2014-04-08 07:30:44
|
<grrrit-wm>
|
('PS1') 'Faidon Liambotis': base: add debian-goodies [operations/puppet] - 'https://gerrit.wikimedia.org/r/124524'
|
|
2014-04-08 07:47:07
|
<_joe|away>
|
!log restarted nginx on cp1044 and cp1043
|
|
2014-04-08 07:47:12
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 07:53:07
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 07:53:07
|
<grrrit-wm>
|
('CR') 'coren': [C: '2'] base: add debian-goodies [operations/puppet] - 'https://gerrit.wikimedia.org/r/124524' (owner: 'Faidon Liambotis')
|
|
2014-04-08 08:02:57
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 08:09:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 08:09:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 08:09:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 08:09:07
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 08:11:47
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 08:15:17
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 08:36:30
|
<siebrand>
|
ori: still working?
|
|
2014-04-08 09:03:47
|
<icinga-wm>
|
PROBLEM - RAID on labstore3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
|
|
2014-04-08 09:04:07
|
<YuviPanda>
|
hashar: help with setting up zuul for the apps? https://gerrit.wikimedia.org/r/#/c/124539/
|
|
2014-04-08 09:08:37
|
<icinga-wm>
|
PROBLEM - Disk space on labstore3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
|
|
2014-04-08 09:08:47
|
<icinga-wm>
|
RECOVERY - RAID on labstore3 is OK: OK: optimal, 12 logical, 12 physical
|
|
2014-04-08 09:08:57
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 09:11:47
|
<icinga-wm>
|
PROBLEM - RAID on labstore3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
|
|
2014-04-08 09:16:55
|
<grrrit-wm>
|
('PS1') 'RobH': Replacing the unified certificate [operations/puppet] - 'https://gerrit.wikimedia.org/r/124542'
|
|
2014-04-08 09:24:34
|
<grrrit-wm>
|
('CR') 'RobH': [C: '2'] Replacing the unified certificate [operations/puppet] - 'https://gerrit.wikimedia.org/r/124542' (owner: 'RobH')
|
|
2014-04-08 09:29:47
|
<icinga-wm>
|
RECOVERY - RAID on labstore3 is OK: OK: optimal, 12 logical, 12 physical
|
|
2014-04-08 09:33:47
|
<icinga-wm>
|
PROBLEM - RAID on labstore3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
|
|
2014-04-08 09:36:37
|
<icinga-wm>
|
RECOVERY - RAID on labstore3 is OK: OK: optimal, 12 logical, 12 physical
|
|
2014-04-08 09:37:37
|
<icinga-wm>
|
RECOVERY - Disk space on labstore3 is OK: DISK OK
|
|
2014-04-08 09:39:19
|
<hashar>
|
YuviPanda: hello
|
|
2014-04-08 09:39:25
|
<YuviPanda>
|
hashar: hello!
|
|
2014-04-08 09:40:00
|
<icinga-wm>
|
PROBLEM - RAID on labstore3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
|
|
2014-04-08 09:40:37
|
<icinga-wm>
|
PROBLEM - Disk space on labstore3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
|
|
2014-04-08 09:40:57
|
<grrrit-wm>
|
('PS1') 'Andrew Bogott': Add eth1 checks to nova compute hosts. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124560'
|
|
2014-04-08 09:44:12
|
<hashar>
|
and we lost YuviPanda
|
|
2014-04-08 09:45:10
|
<sjoerddebruin>
|
Noooo not our panda. :(
|
|
2014-04-08 09:46:25
|
<Steinsplitter>
|
panda \O/
|
|
2014-04-08 09:46:28
|
<icinga-wm>
|
PROBLEM - SSH on labstore3 is CRITICAL: Connection refused
|
|
2014-04-08 09:46:28
|
<icinga-wm>
|
PROBLEM - DPKG on labstore3 is CRITICAL: Connection refused by host
|
|
2014-04-08 09:46:47
|
<icinga-wm>
|
PROBLEM - puppet disabled on labstore3 is CRITICAL: Connection refused by host
|
|
2014-04-08 09:47:00
|
<andrewbogott>
|
mutante: https://gerrit.wikimedia.org/r/#/c/124560/
|
|
2014-04-08 09:47:43
|
<icinga-wm>
|
ACKNOWLEDGEMENT - DPKG on labstore3 is CRITICAL: Connection refused by host daniel_zahn will be decomed - The acknowledgement expires at: 2014-04-09 09:46:44.
|
|
2014-04-08 09:47:44
|
<icinga-wm>
|
ACKNOWLEDGEMENT - Disk space on labstore3 is CRITICAL: Connection refused by host daniel_zahn will be decomed - The acknowledgement expires at: 2014-04-09 09:46:44.
|
|
2014-04-08 09:47:44
|
<icinga-wm>
|
ACKNOWLEDGEMENT - RAID on labstore3 is CRITICAL: Connection refused by host daniel_zahn will be decomed - The acknowledgement expires at: 2014-04-09 09:46:44.
|
|
2014-04-08 09:47:44
|
<icinga-wm>
|
ACKNOWLEDGEMENT - SSH on labstore3 is CRITICAL: Connection refused daniel_zahn will be decomed - The acknowledgement expires at: 2014-04-09 09:46:44.
|
|
2014-04-08 09:47:44
|
<icinga-wm>
|
ACKNOWLEDGEMENT - puppet disabled on labstore3 is CRITICAL: Connection refused by host daniel_zahn will be decomed - The acknowledgement expires at: 2014-04-09 09:46:44.
|
|
2014-04-08 09:49:57
|
<matanya>
|
so nice to see all ops in an europian time zone :)
|
|
2014-04-08 09:50:37
|
<icinga-wm>
|
PROBLEM - Host labstore3 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2014-04-08 09:57:12
|
<grrrit-wm>
|
('CR') 'Dzahn': [C: '-1'] Add eth1 checks to nova compute hosts. ('3' comments) [operations/puppet] - 'https://gerrit.wikimedia.org/r/124560' (owner: 'Andrew Bogott')
|
|
2014-04-08 10:00:49
|
<springle>
|
ori: what is udpprofile::collector, and can i move it from db1014 to... somewhere else?
|
|
2014-04-08 10:02:47
|
<ori>
|
springle: oh, wow. is there any indication that continues to see activity? mediawiki's profiler class can be configured to write to a database, but i didn't know anyone was using it in production. is it not ancient?
|
|
2014-04-08 10:04:56
|
<andrewbogott>
|
mutante, cmjohnson: https://wikitech.wikimedia.org/wiki/Help:Git_rebase#Don.27t_panic
|
|
2014-04-08 10:05:21
|
<thedj>
|
andrewbogott: 42
|
|
2014-04-08 10:05:57
|
<ori>
|
springle: it can go away
|
|
2014-04-08 10:06:34
|
<ori>
|
springle: it was added in this commit: <https://gerrit.wikimedia.org/r/#/c/83953/>;. the message reads: "testing graphite 0.910 on db1014".
|
|
2014-04-08 10:07:04
|
<springle>
|
yeah, asher stole db1014 for graphite
|
|
2014-04-08 10:07:12
|
<springle>
|
trying to steal it back :)
|
|
2014-04-08 10:07:20
|
<springle>
|
ori: thanks
|
|
2014-04-08 10:07:46
|
<ori>
|
springle: it's not in any way implicated in our current graphite setup, which exists solely on tungsten.eqiad.wmnet (and labs)
|
|
2014-04-08 10:08:13
|
<grrrit-wm>
|
('PS2') 'Andrew Bogott': Add eth1 checks to nova compute hosts. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124560'
|
|
2014-04-08 10:08:18
|
<andrewbogott>
|
mutante: ^
|
|
2014-04-08 10:09:24
|
<grrrit-wm>
|
('PS1') 'Cmjohnson': adding ethtool to standard-packages.pp to be able to monitor interface speed [operations/puppet] - 'https://gerrit.wikimedia.org/r/124572'
|
|
2014-04-08 10:11:07
|
<grrrit-wm>
|
('CR') 'jenkins-bot': [V: '-1'] adding ethtool to standard-packages.pp to be able to monitor interface speed [operations/puppet] - 'https://gerrit.wikimedia.org/r/124572' (owner: 'Cmjohnson')
|
|
2014-04-08 10:12:49
|
<grrrit-wm>
|
('CR') 'Dzahn': [C: ''] Add eth1 checks to nova compute hosts. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124560' (owner: 'Andrew Bogott')
|
|
2014-04-08 10:15:34
|
<Jeff_Green>
|
!log update & reboot samarium
|
|
2014-04-08 10:15:38
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 10:15:48
|
<grrrit-wm>
|
('CR') 'Andrew Bogott': [C: '2'] Add eth1 checks to nova compute hosts. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124560' (owner: 'Andrew Bogott')
|
|
2014-04-08 10:16:26
|
<grrrit-wm>
|
('PS1') 'Springle': Remove unused db1014 block. db1014 was renamed tungsten rt5871. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124575'
|
|
2014-04-08 10:18:19
|
<grrrit-wm>
|
('CR') 'Springle': [C: '2'] Remove unused db1014 block. db1014 was renamed tungsten rt5871. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124575' (owner: 'Springle')
|
|
2014-04-08 10:21:04
|
<Jeff_Green>
|
!log update & reboot barium
|
|
2014-04-08 10:21:09
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 10:23:09
|
<grrrit-wm>
|
('PS1') 'Dzahn': add nrpe to base [operations/puppet] - 'https://gerrit.wikimedia.org/r/124576'
|
|
2014-04-08 10:24:10
|
<grrrit-wm>
|
('CR') 'jenkins-bot': [V: '-1'] add nrpe to base [operations/puppet] - 'https://gerrit.wikimedia.org/r/124576' (owner: 'Dzahn')
|
|
2014-04-08 11:09:28
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 11:09:28
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 11:09:28
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 11:09:28
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 11:32:05
|
<grrrit-wm>
|
('PS20') 'Matanya': etherpad: convert into a module [operations/puppet] - 'https://gerrit.wikimedia.org/r/107567'
|
|
2014-04-08 11:32:32
|
<matanya>
|
akosiaris: in a meeting or this ^ can be handled ?
|
|
2014-04-08 11:39:18
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 12:32:58
|
<grrrit-wm>
|
('PS2') 'Dzahn': add nrpe to base [operations/puppet] - 'https://gerrit.wikimedia.org/r/124576'
|
|
2014-04-08 12:39:13
|
<akosiaris>
|
matanya: in ops meeting
|
|
2014-04-08 12:39:19
|
<matanya>
|
sorry
|
|
2014-04-08 12:39:27
|
<akosiaris>
|
and please tell me you did not resubmit from your local repo
|
|
2014-04-08 12:39:48
|
<akosiaris>
|
rebase* sorry
|
|
2014-04-08 12:39:50
|
<grrrit-wm>
|
('PS2') 'Cmjohnson': adding ethtool to standard-packages.pp to be able to monitor interface speed [operations/puppet] - 'https://gerrit.wikimedia.org/r/124572'
|
|
2014-04-08 12:40:26
|
<grrrit-wm>
|
('CR') 'Andrew Bogott': [V: ''] "This looks good -- we'll see if it makes new alarms go off :)" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124576' (owner: 'Dzahn')
|
|
2014-04-08 12:46:38
|
<grrrit-wm>
|
('PS3') 'Cmjohnson': adding ethtool to standard-packages.pp to be able to monitor interface speed [operations/puppet] - 'https://gerrit.wikimedia.org/r/124572'
|
|
2014-04-08 12:48:28
|
<icinga-wm>
|
PROBLEM - DPKG on strontium is CRITICAL: DPKG CRITICAL dpkg reports broken packages
|
|
2014-04-08 12:49:28
|
<icinga-wm>
|
RECOVERY - DPKG on strontium is OK: All packages OK
|
|
2014-04-08 12:49:35
|
<grrrit-wm>
|
('CR') 'Matanya': [C: ''] add nrpe to base [operations/puppet] - 'https://gerrit.wikimedia.org/r/124576' (owner: 'Dzahn')
|
|
2014-04-08 12:50:21
|
<cmjohnson1>
|
paravoid: can you review please https://gerrit.wikimedia.org/r/124572
|
|
2014-04-08 12:50:38
|
<andrewbogott>
|
mutante: https://rt.wikimedia.org/Ticket/Display.html?id=5064
|
|
2014-04-08 12:51:29
|
<grrrit-wm>
|
('CR') 'Dzahn': [C: ''] "yep, if we want to monitor this on everything, then standard-packages sounds good to me" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124572' (owner: 'Cmjohnson')
|
|
2014-04-08 12:52:38
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 12:53:10
|
<grrrit-wm>
|
('CR') 'Alexandros Kosiaris': [C: '2'] adding ethtool to standard-packages.pp to be able to monitor interface speed [operations/puppet] - 'https://gerrit.wikimedia.org/r/124572' (owner: 'Cmjohnson')
|
|
2014-04-08 12:55:34
|
<manybubbles>
|
can anyone around update Elasticsearch in apt?
|
|
2014-04-08 12:55:55
|
<manybubbles>
|
and ack nagios errors (so they don't spam to irc) for a couple horus?
|
|
2014-04-08 12:56:39
|
<logmsgbot>
|
!log reedy updated /a/common to {{Gerrit|Id15ddc665}}: Revert "Group0 wikis to 1.23wmf21"
|
|
2014-04-08 12:56:44
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 12:57:23
|
<grrrit-wm>
|
('PS1') 'Reedy': Non wikipedias to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124591'
|
|
2014-04-08 12:59:03
|
<Reedy>
|
pokes qchris_away and ^d
|
|
2014-04-08 13:01:42
|
<Reedy>
|
Any idea why https://gerrit.wikimedia.org/changes/?q=status:merged+age%3A0d&o=DETAILED_ACCOUNTS&n=100 doesn't work?
|
|
2014-04-08 13:02:00
|
<grrrit-wm>
|
('CR') 'Cmjohnson': [C: '2'] adding ethtool to standard-packages.pp to be able to monitor interface speed [operations/puppet] - 'https://gerrit.wikimedia.org/r/124572' (owner: 'Cmjohnson')
|
|
2014-04-08 13:03:24
|
<Reedy>
|
versus
|
|
2014-04-08 13:03:24
|
<Reedy>
|
http://review.cyanogenmod.org/changes/?q=status:open+age%3A0d&o=DETAILED_ACCOUNTS&n=100
|
|
2014-04-08 13:07:41
|
<grrrit-wm>
|
('PS3') 'Dzahn': add nrpe to base [operations/puppet] - 'https://gerrit.wikimedia.org/r/124576'
|
|
2014-04-08 13:12:48
|
<grrrit-wm>
|
('PS4') 'Dzahn': add nrpe to base [operations/puppet] - 'https://gerrit.wikimedia.org/r/124576'
|
|
2014-04-08 13:15:18
|
<apergos>
|
test
|
|
2014-04-08 13:15:42
|
<apergos>
|
test akosiaris
|
|
2014-04-08 13:15:43
|
<akosiaris>
|
apergos: :-)
|
|
2014-04-08 13:15:51
|
<apergos>
|
manybubbles:
|
|
2014-04-08 13:16:54
|
<mutante>
|
already pinged
|
|
2014-04-08 13:17:06
|
<grrrit-wm>
|
('PS1') 'coren': Tool Labs: forcibly upgrade libssl [operations/puppet] - 'https://gerrit.wikimedia.org/r/124594'
|
|
2014-04-08 13:19:25
|
<grrrit-wm>
|
('CR') 'Dzahn': [C: '2'] "RT #80 :)" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124576' (owner: 'Dzahn')
|
|
2014-04-08 13:21:58
|
<_joe_>
|
ori: If you're here, please let me know :)
|
|
2014-04-08 13:26:57
|
<Reedy>
|
_joe_: Couple of hours from now
|
|
2014-04-08 13:27:05
|
<Reedy>
|
Though, he is around early sometimes
|
|
2014-04-08 13:27:31
|
<_joe_>
|
Reedy: thanks
|
|
2014-04-08 13:30:38
|
<grrrit-wm>
|
('CR') 'RobH': [C: ''] Tool Labs: forcibly upgrade libssl [operations/puppet] - 'https://gerrit.wikimedia.org/r/124594' (owner: 'coren')
|
|
2014-04-08 13:31:20
|
<manybubbles>
|
ottomata: welcome!
|
|
2014-04-08 13:31:34
|
<manybubbles>
|
can you help me get started today?
|
|
2014-04-08 13:31:42
|
<grrrit-wm>
|
('CR') 'coren': [C: '2'] Tool Labs: forcibly upgrade libssl [operations/puppet] - 'https://gerrit.wikimedia.org/r/124594' (owner: 'coren')
|
|
2014-04-08 13:31:50
|
<Reedy>
|
manybubbles: We have an extension for that
|
|
2014-04-08 13:31:51
|
<Reedy>
|
grins
|
|
2014-04-08 13:31:57
|
<manybubbles>
|
Reedy: thanks!
|
|
2014-04-08 13:32:01
|
<manybubbles>
|
I totally used it a while ago
|
|
2014-04-08 13:32:27
|
<qchris_away>
|
Reedy: Because we're using /r/ to mark the reverse proxy ...
|
|
2014-04-08 13:32:33
|
<qchris_away>
|
Reedy: https://gerrit.wikimedia.org/r/changes/?q=status:merged+age%3A0d&o=DETAILED_ACCOUNTS&n=100
|
|
2014-04-08 13:32:37
|
<qchris_away>
|
Reedy: ^ should work
|
|
2014-04-08 13:32:47
|
<Reedy>
|
Aha, sweet!
|
|
2014-04-08 13:33:43
|
<grrrit-wm>
|
('PS1') 'RobH': replace blog.wikimedia.org certificate [operations/puppet] - 'https://gerrit.wikimedia.org/r/124595'
|
|
2014-04-08 13:35:07
|
<manybubbles>
|
ottomata: I need Elasticsearch 1.1.0 shoved into apt
|
|
2014-04-08 13:35:37
|
<grrrit-wm>
|
('PS2') 'RobH': replace blog.wikimedia.org certificate [operations/puppet] - 'https://gerrit.wikimedia.org/r/124595'
|
|
2014-04-08 13:36:15
|
<Reedy>
|
qchris: thanks
|
|
2014-04-08 13:36:22
|
<qchris>
|
yw
|
|
2014-04-08 13:37:04
|
<icinga-wm>
|
PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds
|
|
2014-04-08 13:37:33
|
<mutante>
|
!log restarting gitblit
|
|
2014-04-08 13:37:33
|
<grrrit-wm>
|
('CR') 'RobH': [C: '2'] replace blog.wikimedia.org certificate [operations/puppet] - 'https://gerrit.wikimedia.org/r/124595' (owner: 'RobH')
|
|
2014-04-08 13:37:37
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 13:39:00
|
<RobH>
|
!log replacing the blog cert, if holmium crashes I didn't do it correctly.
|
|
2014-04-08 13:39:01
|
<grrrit-wm>
|
('PS1') 'Faidon Liambotis': Revert "Giving Nik shell access to analytics1004 to do some elasticsearch load testing" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124597'
|
|
2014-04-08 13:39:03
|
<ottomata>
|
manybubbles: ok!
|
|
2014-04-08 13:39:03
|
<morebots>
|
Logged the message, RobH
|
|
2014-04-08 13:39:04
|
<icinga-wm>
|
RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 305803 bytes in 9.337 second response time
|
|
2014-04-08 13:39:08
|
<manybubbles>
|
thanks!
|
|
2014-04-08 13:39:28
|
<Jeff_Green>
|
!log update & reboot tellurium
|
|
2014-04-08 13:39:33
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 13:39:47
|
<grrrit-wm>
|
('CR') 'jenkins-bot': [V: '-1'] Revert "Giving Nik shell access to analytics1004 to do some elasticsearch load testing" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124597' (owner: 'Faidon Liambotis')
|
|
2014-04-08 13:41:14
|
<icinga-wm>
|
PROBLEM - Host tellurium is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2014-04-08 13:42:38
|
<grrrit-wm>
|
('PS2') 'Faidon Liambotis': Revert "Giving Nik shell access to analytics1004 to do some elasticsearch load testing" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124597'
|
|
2014-04-08 13:43:27
|
<grrrit-wm>
|
('CR') 'Faidon Liambotis': [C: '2' V: '2'] Revert "Giving Nik shell access to analytics1004 to do some elasticsearch load testing" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124597' (owner: 'Faidon Liambotis')
|
|
2014-04-08 13:44:28
|
<grrrit-wm>
|
('CR') 'Manybubbles': "Is there a better place to run this?" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124597' (owner: 'Faidon Liambotis')
|
|
2014-04-08 13:45:14
|
<icinga-wm>
|
RECOVERY - Host tellurium is UP: PING OK - Packet loss = 0%, RTA = 1.11 ms
|
|
2014-04-08 13:46:13
|
<RobH>
|
!log upgraded libssl on holmium
|
|
2014-04-08 13:46:18
|
<morebots>
|
Logged the message, RobH
|
|
2014-04-08 13:48:49
|
<paravoid>
|
ottomata: kafka upgrade doesn't work on an1004
|
|
2014-04-08 13:49:41
|
<ottomata>
|
paravoid, analytics1004 (and analytics1003) were kafka test brokers, and were never productionized or puppetized
|
|
2014-04-08 13:49:50
|
<ottomata>
|
i thought I had removed kafka from analytics1004, actually
|
|
2014-04-08 13:50:38
|
<manybubbles>
|
ottomata: can you install git fat on tin?
|
|
2014-04-08 13:50:42
|
<manybubbles>
|
I cannot
|
|
2014-04-08 13:50:46
|
<ottomata>
|
hm, sure, why do you need git-fat there?
|
|
2014-04-08 13:50:55
|
<manybubbles>
|
to git deploy
|
|
2014-04-08 13:50:58
|
<manybubbles>
|
to Elasticsearch
|
|
2014-04-08 13:51:07
|
<manybubbles>
|
the plugins
|
|
2014-04-08 13:51:14
|
<manybubbles>
|
or is there another server
|
|
2014-04-08 13:51:17
|
<ottomata>
|
you don't need git-fat on tin though
|
|
2014-04-08 13:51:23
|
<ottomata>
|
the git-fat commands are run on deplo hsots
|
|
2014-04-08 13:51:27
|
<ottomata>
|
on the targets
|
|
2014-04-08 13:51:46
|
<manybubbles>
|
huh, I'm used to running it on the server to check the jars got there. I'll just do it without and see
|
|
2014-04-08 13:53:21
|
<manybubbles>
|
ottomata: that worked as you said it would
|
|
2014-04-08 13:53:35
|
<manybubbles>
|
!log synced first Elasticsearch plugin to production Elasticsearch servers
|
|
2014-04-08 13:53:39
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 13:54:01
|
<manybubbles>
|
!log they'll pick it up during the rolling restart today to upgrade to 1.1.0
|
|
2014-04-08 13:54:05
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 13:54:08
|
<ottomata>
|
cool
|
|
2014-04-08 13:54:18
|
<ottomata>
|
manybubbles: , i was going to start reinstalling an elastic search server today
|
|
2014-04-08 13:54:33
|
<manybubbles>
|
ottomata: not a _great_ day for it
|
|
2014-04-08 13:54:37
|
<manybubbles>
|
because I'm upgrading to 1.1.0
|
|
2014-04-08 13:54:43
|
<ottomata>
|
ok
|
|
2014-04-08 13:54:45
|
<manybubbles>
|
that is on the deployment calendar and everything
|
|
2014-04-08 13:55:05
|
<manybubbles>
|
maybe tomorrow?
|
|
2014-04-08 13:57:09
|
<ottomata>
|
sure
|
|
2014-04-08 14:04:07
|
<manybubbles>
|
ottomata: please ping me when you get a chance to update apt
|
|
2014-04-08 14:04:35
|
<ottomata>
|
i was about to to do it, but am in standup now
|
|
2014-04-08 14:04:36
|
<ottomata>
|
um
|
|
2014-04-08 14:04:41
|
<ottomata>
|
q for akosiaris, if you are around
|
|
2014-04-08 14:04:54
|
<ottomata>
|
I should change VerifyRelease, right?
|
|
2014-04-08 14:04:54
|
<icinga-wm>
|
PROBLEM - DPKG on labstore4 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
|
|
2014-04-08 14:04:59
|
<ottomata>
|
i'm trying to find the right thing to change it to
|
|
2014-04-08 14:05:14
|
<ottomata>
|
i downloaded 1.1's Release.gpg and am doing what the reprepro man page says to do
|
|
2014-04-08 14:05:17
|
<ottomata>
|
but am not sure
|
|
2014-04-08 14:05:23
|
<ottomata>
|
the output doesn't look like what you have
|
|
2014-04-08 14:05:54
|
<icinga-wm>
|
RECOVERY - DPKG on labstore4 is OK: All packages OK
|
|
2014-04-08 14:09:44
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 14:09:44
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 14:09:44
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 14:09:44
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 14:11:17
|
<grrrit-wm>
|
('PS1') 'Andrew Bogott': Install and use check_ssl_cert tool to validate certs. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124601'
|
|
2014-04-08 14:18:13
|
<grrrit-wm>
|
('PS2') 'Andrew Bogott': Install and use check_ssl_cert tool to validate certs. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124601'
|
|
2014-04-08 14:19:21
|
<grrrit-wm>
|
('PS1') 'Ottomata': reprepro/updates - upgrading elasticsearch to 1.1 [operations/puppet] - 'https://gerrit.wikimedia.org/r/124603'
|
|
2014-04-08 14:20:08
|
<grrrit-wm>
|
('CR') 'Ottomata': [C: '2' V: '2'] reprepro/updates - upgrading elasticsearch to 1.1 [operations/puppet] - 'https://gerrit.wikimedia.org/r/124603' (owner: 'Ottomata')
|
|
2014-04-08 14:23:54
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1002 is CRITICAL: Connection refused
|
|
2014-04-08 14:24:06
|
<ottomata>
|
manybubbles: http://apt.wikimedia.org/wikimedia/pool/main/e/elasticsearch/
|
|
2014-04-08 14:24:09
|
<ottomata>
|
look ok?
|
|
2014-04-08 14:28:54
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1002 is OK: OK - Certificate will expire on 01/20/2016 12:00.
|
|
2014-04-08 14:29:45
|
<manybubbles>
|
ottomata: looks good - let me try elastic1001
|
|
2014-04-08 14:30:35
|
<grrrit-wm>
|
('PS3') 'Andrew Bogott': Install and use check_ssl_cert tool to validate certs. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124601'
|
|
2014-04-08 14:30:57
|
<andrewbogott>
|
mutante, ^ pls?
|
|
2014-04-08 14:31:37
|
<manybubbles>
|
!log upgrading elastic1001
|
|
2014-04-08 14:31:42
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 14:32:38
|
<manybubbles>
|
!log woops, just restarted elastic1002. silly me
|
|
2014-04-08 14:32:42
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 14:32:46
|
<manybubbles>
|
!log no harm done, just lost time
|
|
2014-04-08 14:32:50
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 14:33:53
|
<manybubbles>
|
ottomata: can you make nagios not bother us about Elasticsearch warning over the next few hours?
|
|
2014-04-08 14:33:56
|
<manybubbles>
|
I'm paying attention
|
|
2014-04-08 14:34:25
|
<ottomata>
|
uh hm
|
|
2014-04-08 14:35:43
|
<ottomata>
|
i think so, how long manybubbles
|
|
2014-04-08 14:35:45
|
<ottomata>
|
4 hours?
|
|
2014-04-08 14:35:48
|
<manybubbles>
|
sure!
|
|
2014-04-08 14:36:14
|
<icinga-wm>
|
PROBLEM - NTP peers on linne is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown
|
|
2014-04-08 14:38:14
|
<icinga-wm>
|
RECOVERY - NTP peers on linne is OK: NTP OK: Offset 0.016747 secs
|
|
2014-04-08 14:44:43
|
<mutante>
|
andrewbogott: https://gerrit.wikimedia.org/r/#/c/77332/7/modules/base/manifests/monitoring/host.pp
|
|
2014-04-08 14:44:51
|
<grrrit-wm>
|
('PS4') 'Andrew Bogott': Install and use check_ssl_cert tool to validate certs. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124601'
|
|
2014-04-08 14:54:18
|
<grrrit-wm>
|
('PS5') 'Andrew Bogott': Install and use check_ssl_cert tool to validate certs. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124601'
|
|
2014-04-08 14:54:59
|
<grrrit-wm>
|
('PS3') 'Cmjohnson': add interface speed check for all hosts [operations/puppet] - 'https://gerrit.wikimedia.org/r/124606'
|
|
2014-04-08 15:01:42
|
<cmjohnson>
|
mutante: can you review https://gerrit.wikimedia.org/r/124606
|
|
2014-04-08 15:02:06
|
<grrrit-wm>
|
('CR') 'Alexandros Kosiaris': [C: '-1'] "Great idea. Minor stuff here and there like making it parameterizable but looks nice." ('6' comments) [operations/puppet] - 'https://gerrit.wikimedia.org/r/124606' (owner: 'Cmjohnson')
|
|
2014-04-08 15:03:10
|
<ottomata>
|
manybubbles: i think I just scheduled downtime in icinga for elastic search for the next ~4 hours
|
|
2014-04-08 15:03:19
|
<ottomata>
|
never done that before, so not sure what it will do
|
|
2014-04-08 15:03:47
|
<grrrit-wm>
|
('PS1') 'Rush': module to manage new python-diamond package [operations/puppet] - 'https://gerrit.wikimedia.org/r/124608'
|
|
2014-04-08 15:04:54
|
<manybubbles>
|
ottomata: its cool!
|
|
2014-04-08 15:04:56
|
<manybubbles>
|
thanks
|
|
2014-04-08 15:07:45
|
<grrrit-wm>
|
('CR') 'Ottomata': module to manage new python-diamond package ('5' comments) [operations/puppet] - 'https://gerrit.wikimedia.org/r/124608' (owner: 'Rush')
|
|
2014-04-08 15:08:18
|
<grrrit-wm>
|
('CR') 'Dzahn': [C: ''] Install and use check_ssl_cert tool to validate certs. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124601' (owner: 'Andrew Bogott')
|
|
2014-04-08 15:12:34
|
<grrrit-wm>
|
('PS2') 'Rush': module to manage new python-diamond package [operations/puppet] - 'https://gerrit.wikimedia.org/r/124608'
|
|
2014-04-08 15:13:35
|
<grrrit-wm>
|
('CR') 'jenkins-bot': [V: '-1'] module to manage new python-diamond package [operations/puppet] - 'https://gerrit.wikimedia.org/r/124608' (owner: 'Rush')
|
|
2014-04-08 15:15:36
|
<grrrit-wm>
|
('PS3') 'Rush': module to manage new python-diamond package [operations/puppet] - 'https://gerrit.wikimedia.org/r/124608'
|
|
2014-04-08 15:16:34
|
<icinga-wm>
|
PROBLEM - Host virt1000 is DOWN: CRITICAL - Host Unreachable (208.80.154.18)
|
|
2014-04-08 15:16:42
|
<RobH>
|
!log all ssl servers in eqiad have been updated with new cert and restarted
|
|
2014-04-08 15:16:51
|
<RobH>
|
!log rolling updates on ssl3001-3003 presently
|
|
2014-04-08 15:17:10
|
<grrrit-wm>
|
('PS1') 'Dzahn': enable base monitoring for ALL hosts [operations/puppet] - 'https://gerrit.wikimedia.org/r/124609'
|
|
2014-04-08 15:17:24
|
<icinga-wm>
|
PROBLEM - Host labs-ns1.wikimedia.org is DOWN: CRITICAL - Host Unreachable (208.80.154.19)
|
|
2014-04-08 15:18:04
|
<icinga-wm>
|
RECOVERY - Host virt1000 is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms
|
|
2014-04-08 15:19:03
|
<grrrit-wm>
|
('CR') 'Andrew Bogott': [C: '2'] Install and use check_ssl_cert tool to validate certs. [operations/puppet] - 'https://gerrit.wikimedia.org/r/124601' (owner: 'Andrew Bogott')
|
|
2014-04-08 15:19:04
|
<icinga-wm>
|
RECOVERY - Host labs-ns1.wikimedia.org is UP: PING OK - Packet loss = 0%, RTA = 0.98 ms
|
|
2014-04-08 15:19:07
|
<mutante>
|
apergos: https://gerrit.wikimedia.org/r/#/c/124609/1
|
|
2014-04-08 15:19:46
|
<mutante>
|
ugly, eh.. since i have to change all those lines because of indentation :p
|
|
2014-04-08 15:22:25
|
<grrrit-wm>
|
('CR') 'ArielGlenn': [C: ''] enable base monitoring for ALL hosts [operations/puppet] - 'https://gerrit.wikimedia.org/r/124609' (owner: 'Dzahn')
|
|
2014-04-08 15:22:39
|
<grrrit-wm>
|
('CR') 'Dzahn': [C: '2'] enable base monitoring for ALL hosts [operations/puppet] - 'https://gerrit.wikimedia.org/r/124609' (owner: 'Dzahn')
|
|
2014-04-08 15:23:46
|
<grrrit-wm>
|
('CR') 'Ottomata': module to manage new python-diamond package ('2' comments) [operations/puppet] - 'https://gerrit.wikimedia.org/r/124608' (owner: 'Rush')
|
|
2014-04-08 15:27:31
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4009 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:41
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl3003 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:41
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1006 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:41
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4014 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:51
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1004 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:51
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1005 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:51
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4008 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:51
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4004 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:51
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4015 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:52
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4001 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:52
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4017 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:53
|
<icinga-wm>
|
PROBLEM - HTTPS on amssq47 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:53
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1002 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:54
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1001 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:54
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4005 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:27:55
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4012 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:01
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4016 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:01
|
<icinga-wm>
|
PROBLEM - HTTPS on sodium is CRITICAL: SSL_CERT CRITICAL lists.wikimedia.org: invalid CN (lists.wikimedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:11
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1007 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:11
|
<icinga-wm>
|
PROBLEM - HTTPS on iodine is CRITICAL: SSL_CERT CRITICAL ticket.wikimedia.org: invalid CN (ticket.wikimedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:11
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl3002 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:11
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl3001 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:11
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4018 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:12
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1008 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:12
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1009 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:13
|
<icinga-wm>
|
PROBLEM - HTTPS on ssl1003 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:13
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4013 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:14
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4003 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:14
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4007 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:15
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4011 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:15
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4010 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:21
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4020 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:21
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4006 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:31
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4002 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:28:31
|
<icinga-wm>
|
PROBLEM - HTTPS on cp4019 is CRITICAL: SSL_CERT CRITICAL *.wikipedia.org: invalid CN (*.wikipedia.org does not match *.wikimedia.org)
|
|
2014-04-08 15:30:02
|
<greg-g>
|
holy fun :)
|
|
2014-04-08 15:30:37
|
<aude>
|
:o
|
|
2014-04-08 15:32:08
|
<greg-g>
|
aude: getting to your email :)
|
|
2014-04-08 15:32:13
|
<aude>
|
ok
|
|
2014-04-08 15:32:25
|
<aude>
|
want to see if it's ok to do today
|
|
2014-04-08 15:32:35
|
<aude>
|
anytime works for us, i suppose
|
|
2014-04-08 15:34:45
|
<greg-g>
|
aude: tl;dr of email: yep, looks good
|
|
2014-04-08 15:34:50
|
<aude>
|
ok
|
|
2014-04-08 15:35:07
|
<aude>
|
we were smart to put i18n stuff a while ago :)
|
|
2014-04-08 15:35:42
|
<icinga-wm>
|
PROBLEM - RAID on holmium is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded)
|
|
2014-04-08 15:35:52
|
<icinga-wm>
|
PROBLEM - DPKG on fenari is CRITICAL: NRPE: Command check_dpkg not defined
|
|
2014-04-08 15:36:01
|
<andrewbogott>
|
the https failures are me muching with monitoring, nothing to worry about
|
|
2014-04-08 15:36:02
|
<icinga-wm>
|
PROBLEM - Disk space on fenari is CRITICAL: NRPE: Command check_disk_space not defined
|
|
2014-04-08 15:36:12
|
<icinga-wm>
|
PROBLEM - RAID on fenari is CRITICAL: NRPE: Command check_raid not defined
|
|
2014-04-08 15:36:22
|
<icinga-wm>
|
PROBLEM - puppet disabled on fenari is CRITICAL: NRPE: Command check_puppet_disabled not defined
|
|
2014-04-08 15:36:57
|
<hashar>
|
mutante: fenari is not happy :-D
|
|
2014-04-08 15:38:21
|
<mutante>
|
hashar: thanks, that's cause we just added more monitoring
|
|
2014-04-08 15:38:33
|
<mutante>
|
RT #80 :)
|
|
2014-04-08 15:38:48
|
<hashar>
|
mutante: yeah I noticed your puppet change. Guess fenari is missing some bits
|
|
2014-04-08 15:41:12
|
<mutante>
|
hashar: wasn't running nagios-nrpe-server
|
|
2014-04-08 15:41:52
|
<mutante>
|
greg-g: re: SSL certs, andrewbogott is on that one
|
|
2014-04-08 15:41:57
|
<mutante>
|
ops monitoring sprint over here
|
|
2014-04-08 15:42:11
|
<greg-g>
|
mutante: ahh, good to know who's on point for that, thanks
|
|
2014-04-08 15:42:23
|
<greg-g>
|
wasn't sure if it'd be a opsen party thing or not
|
|
2014-04-08 15:42:44
|
<mutante>
|
it is. ops in Athens
|
|
2014-04-08 15:43:05
|
<mutante>
|
that check is new, in that it checks for validity of cert, not just expiry
|
|
2014-04-08 15:43:18
|
<mutante>
|
and wikimedia vs. wikipedia thing
|
|
2014-04-08 15:43:30
|
<greg-g>
|
nods
|
|
2014-04-08 15:44:52
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 438.266663
|
|
2014-04-08 15:45:02
|
<grrrit-wm>
|
('PS1') 'Andrew Bogott': When checking unified certs, check for *.wikipedia.org [operations/puppet] - 'https://gerrit.wikimedia.org/r/124616'
|
|
2014-04-08 15:45:32
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 434.533325
|
|
2014-04-08 15:46:21
|
<grrrit-wm>
|
('CR') 'Andrew Bogott': [C: '2'] When checking unified certs, check for *.wikipedia.org [operations/puppet] - 'https://gerrit.wikimedia.org/r/124616' (owner: 'Andrew Bogott')
|
|
2014-04-08 15:46:22
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 12:45:20 PM UTC
|
|
2014-04-08 15:53:10
|
<icinga-wm>
|
RECOVERY - RAID on fenari is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0
|
|
2014-04-08 15:53:17
|
<mutante>
|
hashar: ^ :)
|
|
2014-04-08 15:53:20
|
<icinga-wm>
|
RECOVERY - puppet disabled on fenari is OK: OK
|
|
2014-04-08 15:53:26
|
<hashar>
|
nice
|
|
2014-04-08 15:53:40
|
<icinga-wm>
|
RECOVERY - Disk space on fenari is OK: DISK OK
|
|
2014-04-08 15:53:41
|
<mutante>
|
RT #80 ftw
|
|
2014-04-08 15:53:48
|
<andrewbogott>
|
With any luck there'll be another flood of OKs in a minute...
|
|
2014-04-08 15:53:50
|
<icinga-wm>
|
RECOVERY - DPKG on fenari is OK: All packages OK
|
|
2014-04-08 15:54:10
|
<icinga-wm>
|
PROBLEM - puppet disabled on bast1001 is CRITICAL: NRPE: Command check_puppet_disabled not defined
|
|
2014-04-08 15:54:10
|
<icinga-wm>
|
PROBLEM - Disk space on cp3003 is CRITICAL: NRPE: Command check_disk_space not defined
|
|
2014-04-08 15:54:10
|
<icinga-wm>
|
PROBLEM - Disk space on dobson is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:10
|
<icinga-wm>
|
PROBLEM - DPKG on pdf2 is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:20
|
<icinga-wm>
|
PROBLEM - puppet disabled on iron is CRITICAL: NRPE: Command check_puppet_disabled not defined
|
|
2014-04-08 15:54:20
|
<icinga-wm>
|
PROBLEM - RAID on dobson is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:20
|
<icinga-wm>
|
PROBLEM - RAID on cp3003 is CRITICAL: NRPE: Command check_raid not defined
|
|
2014-04-08 15:54:20
|
<icinga-wm>
|
PROBLEM - Disk space on pdf2 is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:30
|
<icinga-wm>
|
PROBLEM - puppet disabled on dobson is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:30
|
<icinga-wm>
|
PROBLEM - RAID on pdf2 is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:30
|
<icinga-wm>
|
PROBLEM - DPKG on iodine is CRITICAL: NRPE: Command check_dpkg not defined
|
|
2014-04-08 15:54:30
|
<icinga-wm>
|
PROBLEM - puppet disabled on pdf2 is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:40
|
<icinga-wm>
|
PROBLEM - Disk space on iodine is CRITICAL: NRPE: Command check_disk_space not defined
|
|
2014-04-08 15:54:40
|
<icinga-wm>
|
PROBLEM - puppet disabled on cp3003 is CRITICAL: NRPE: Command check_puppet_disabled not defined
|
|
2014-04-08 15:54:40
|
<icinga-wm>
|
PROBLEM - DPKG on pdf3 is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:48
|
<andrewbogott>
|
that's not what I meant
|
|
2014-04-08 15:54:50
|
<icinga-wm>
|
PROBLEM - RAID on iodine is CRITICAL: NRPE: Command check_raid not defined
|
|
2014-04-08 15:54:50
|
<icinga-wm>
|
PROBLEM - Disk space on pdf3 is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:50
|
<icinga-wm>
|
PROBLEM - DPKG on tridge is CRITICAL: NRPE: Command check_dpkg not defined
|
|
2014-04-08 15:54:50
|
<icinga-wm>
|
PROBLEM - DPKG on bast1001 is CRITICAL: NRPE: Command check_dpkg not defined
|
|
2014-04-08 15:54:51
|
<icinga-wm>
|
PROBLEM - puppet disabled on iodine is CRITICAL: NRPE: Command check_puppet_disabled not defined
|
|
2014-04-08 15:54:51
|
<icinga-wm>
|
PROBLEM - RAID on pdf3 is CRITICAL: Connection refused by host
|
|
2014-04-08 15:54:51
|
<icinga-wm>
|
PROBLEM - Disk space on tridge is CRITICAL: NRPE: Command check_disk_space not defined
|
|
2014-04-08 15:55:00
|
<icinga-wm>
|
PROBLEM - Disk space on bast1001 is CRITICAL: NRPE: Command check_disk_space not defined
|
|
2014-04-08 15:55:00
|
<icinga-wm>
|
PROBLEM - puppet disabled on pdf3 is CRITICAL: Connection refused by host
|
|
2014-04-08 15:55:10
|
<icinga-wm>
|
PROBLEM - Disk space on iron is CRITICAL: NRPE: Command check_disk_space not defined
|
|
2014-04-08 15:55:10
|
<icinga-wm>
|
PROBLEM - RAID on bast1001 is CRITICAL: NRPE: Command check_raid not defined
|
|
2014-04-08 15:55:10
|
<icinga-wm>
|
PROBLEM - DPKG on dobson is CRITICAL: Connection refused by host
|
|
2014-04-08 15:55:10
|
<icinga-wm>
|
PROBLEM - DPKG on cp3003 is CRITICAL: NRPE: Command check_dpkg not defined
|
|
2014-04-08 15:55:10
|
<icinga-wm>
|
PROBLEM - DPKG on virt1000 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
|
|
2014-04-08 15:55:10
|
<icinga-wm>
|
PROBLEM - puppet disabled on tridge is CRITICAL: NRPE: Command check_puppet_disabled not defined
|
|
2014-04-08 15:55:41
|
<greg-g>
|
ahhh, so today is going to be a worthless -operations channel day, more than normal, due to the sprint? :)
|
|
2014-04-08 15:56:03
|
<andrewbogott>
|
We're about to all go to dinner though.
|
|
2014-04-08 15:56:09
|
<andrewbogott>
|
So things should quiet down shortly.
|
|
2014-04-08 15:56:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 12:55:50 PM UTC
|
|
2014-04-08 15:56:19
|
<andrewbogott>
|
But the channel will still be useless if you want to talk to ops :)
|
|
2014-04-08 15:56:50
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 15:57:03
|
<mutante>
|
will start nagios-nrpe-server on those
|
|
2014-04-08 15:57:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 12:56:15 PM UTC
|
|
2014-04-08 15:58:42
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl3001 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 15:58:42
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1006 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 15:58:52
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1007 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 15:58:52
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1002 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 15:59:32
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 15:59:52
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 16:00:04
|
<aude>
|
back in 5 min or so
|
|
2014-04-08 16:00:06
|
<grrrit-wm>
|
('Abandoned') 'Physikerwelt': WIP: Enable orthogonal MathJax config [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/110240' (owner: 'Physikerwelt')
|
|
2014-04-08 16:00:42
|
<icinga-wm>
|
PROBLEM - DPKG on mchenry is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake.
|
|
2014-04-08 16:00:42
|
<icinga-wm>
|
PROBLEM - Disk space on mchenry is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake.
|
|
2014-04-08 16:00:52
|
<icinga-wm>
|
PROBLEM - RAID on mchenry is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake.
|
|
2014-04-08 16:01:02
|
<icinga-wm>
|
PROBLEM - puppet disabled on mchenry is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake.
|
|
2014-04-08 16:02:22
|
<icinga-wm>
|
PROBLEM - Puppet freshness on ms6 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:02:03 PM UTC
|
|
2014-04-08 16:04:37
|
<aude>
|
back
|
|
2014-04-08 16:08:22
|
<icinga-wm>
|
PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:07:31 PM UTC
|
|
2014-04-08 16:09:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:09:07 PM UTC
|
|
2014-04-08 16:09:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs4003 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:08:32 PM UTC
|
|
2014-04-08 16:09:27
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4020 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:09:27
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4006 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:09:27
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4013 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:09:37
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4009 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:09:37
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4010 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:09:37
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl3003 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:09:47
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl3002 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:09:47
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1004 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:09:56
|
<paravoid>
|
ottomata: ping
|
|
2014-04-08 16:09:57
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4012 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:10:07
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4016 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:10:07
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1008 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:10:07
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 16:10:07
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4018 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:10:17
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1009 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:11:23
|
<paravoid>
|
ottomata: ping ping
|
|
2014-04-08 16:12:47
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 16:12:49
|
<ottomata>
|
pong pong
|
|
2014-04-08 16:13:05
|
<ottomata>
|
paravoid
|
|
2014-04-08 16:13:08
|
<ottomata>
|
wassupp
|
|
2014-04-08 16:13:14
|
<paravoid>
|
what's with stat1's puppet?
|
|
2014-04-08 16:13:18
|
<paravoid>
|
why is it admin disabled?
|
|
2014-04-08 16:13:47
|
<ottomata>
|
because it is going to be decomed very soon
|
|
2014-04-08 16:13:56
|
<ottomata>
|
and i wanted to make puppet changes that would apply to stat1003 but not mess with what was on stat1
|
|
2014-04-08 16:14:05
|
<ottomata>
|
and I didn't want to re-write a bunch of statistics.pp stuff :/
|
|
2014-04-08 16:14:07
|
<_joe_>
|
ori: are you around? seems like graphite is *not* working
|
|
2014-04-08 16:14:24
|
<paravoid>
|
ottomata: that's bad
|
|
2014-04-08 16:14:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:13:54 PM UTC
|
|
2014-04-08 16:14:35
|
<ottomata>
|
paravoid: even if we are going to decom it soon?
|
|
2014-04-08 16:14:36
|
<paravoid>
|
ottomata: can you remove the "include statistics*" stuff and enable it again?
|
|
2014-04-08 16:14:40
|
<paravoid>
|
yes
|
|
2014-04-08 16:14:42
|
<ottomata>
|
yeah probably can
|
|
2014-04-08 16:14:47
|
<paravoid>
|
because it's messing with monitoring and all that
|
|
2014-04-08 16:15:06
|
<ottomata>
|
ah i see it
|
|
2014-04-08 16:15:20
|
<ottomata>
|
paravoid, what is the differnece between the 3 numbers in each severity category in icinga?
|
|
2014-04-08 16:15:25
|
<mark>
|
ottomata: disabling puppet for more than a few hours max is almost always a really bad idea
|
|
2014-04-08 16:15:31
|
<ottomata>
|
mark, ok, noted.
|
|
2014-04-08 16:15:36
|
<mark>
|
thanks
|
|
2014-04-08 16:16:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:16:04 PM UTC
|
|
2014-04-08 16:16:27
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 16:17:07
|
<_joe_>
|
:/
|
|
2014-04-08 16:17:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on dataset1001 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:16:39 PM UTC
|
|
2014-04-08 16:18:10
|
<ottomata>
|
mark, can you help with the current network ACL problems?
|
|
2014-04-08 16:18:22
|
<mark>
|
sorry, what's that?
|
|
2014-04-08 16:18:25
|
<ottomata>
|
analytics nodes can't talk to apt
|
|
2014-04-08 16:18:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:17:50 PM UTC
|
|
2014-04-08 16:18:30
|
<ottomata>
|
nor statsd.eqiad.wmnet
|
|
2014-04-08 16:18:32
|
<ottomata>
|
https://rt.wikimedia.org/Ticket/Display.html?id=4433
|
|
2014-04-08 16:18:37
|
<ottomata>
|
I added to the bottom of that ticket
|
|
2014-04-08 16:18:51
|
<mark>
|
ok
|
|
2014-04-08 16:18:59
|
<ottomata>
|
i think vanadium was having the same trouble, is it on the vlan too?
|
|
2014-04-08 16:19:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:19:10 PM UTC
|
|
2014-04-08 16:19:31
|
<aude>
|
still working on wikiquote
|
|
2014-04-08 16:19:35
|
<mark>
|
we can look at getting rid of those ACLs perhaps
|
|
2014-04-08 16:19:41
|
<mark>
|
but we'll need to discuss what you're doing with firewalling
|
|
2014-04-08 16:20:18
|
<grrrit-wm>
|
('PS1') 'Ottomata': Disabling statistics roles on stat1 [operations/puppet] - 'https://gerrit.wikimedia.org/r/124621'
|
|
2014-04-08 16:20:18
|
<se4598>
|
the fingerprint of the wikis SSL cert apparently changed, but it is not a new issued cert but with the same dates as the previous one that i saved. Is that okay that the fingerprint changed?
|
|
2014-04-08 16:20:34
|
<ottomata>
|
mark, yeah, hm, not sure, i kind of like them
|
|
2014-04-08 16:20:35
|
<paravoid>
|
se4598: yes
|
|
2014-04-08 16:20:45
|
<ottomata>
|
especially since anyone with hadoop access can launch whatever mapreduce jobs they want
|
|
2014-04-08 16:21:37
|
<grrrit-wm>
|
('CR') 'Ottomata': [C: '2' V: '2'] Disabling statistics roles on stat1 [operations/puppet] - 'https://gerrit.wikimedia.org/r/124621' (owner: 'Ottomata')
|
|
2014-04-08 16:21:37
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 16:21:44
|
<ottomata>
|
hmmmm
|
|
2014-04-08 16:21:48
|
<ottomata>
|
that's weird
|
|
2014-04-08 16:21:59
|
<ottomata>
|
checking on that 5xx thing in a sec
|
|
2014-04-08 16:22:05
|
<ottomata>
|
that's surely my fault...
|
|
2014-04-08 16:22:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:21:21 PM UTC
|
|
2014-04-08 16:22:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs1001 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:21:26 PM UTC
|
|
2014-04-08 16:22:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs4002 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:22:07 PM UTC
|
|
2014-04-08 16:22:53
|
<ottomata>
|
hmm, graphite down?
|
|
2014-04-08 16:23:04
|
<mark>
|
ottomata: statsd access for analytics seems already there
|
|
2014-04-08 16:23:07
|
<ottomata>
|
maybe that 5xx thing is not my fault!
|
|
2014-04-08 16:23:26
|
<ottomata>
|
yeah, mark, i think we already had these set up too
|
|
2014-04-08 16:23:27
|
<icinga-wm>
|
PROBLEM - Puppet freshness on virt2 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:22:28 PM UTC
|
|
2014-04-08 16:23:37
|
<icinga-wm>
|
RECOVERY - Puppet freshness on stat1 is OK: puppet ran at Tue Apr 8 16:23:30 UTC 2014
|
|
2014-04-08 16:23:43
|
<ottomata>
|
but it seems that they aren't working right now, starting yesterday when I tried
|
|
2014-04-08 16:24:02
|
<grrrit-wm>
|
('PS1') 'Hashar': beta: reenable fatalmonitor script on eqiad [operations/puppet] - 'https://gerrit.wikimedia.org/r/124624'
|
|
2014-04-08 16:24:13
|
<mark>
|
and carbon is in there already too
|
|
2014-04-08 16:24:15
|
<ottomata>
|
mark, unless pings just aren't allowed and i'm checking wrong?
|
|
2014-04-08 16:24:24
|
<mark>
|
pings may not be allowed no
|
|
2014-04-08 16:24:27
|
<ottomata>
|
ori and I both had trouble runnign apt-get update because we coudln't talk to carbon
|
|
2014-04-08 16:24:31
|
<mark>
|
check again?
|
|
2014-04-08 16:24:35
|
<ottomata>
|
yeah checking
|
|
2014-04-08 16:24:48
|
<ottomata>
|
and i was trying to run sqstat on analytics1003
|
|
2014-04-08 16:24:52
|
<ottomata>
|
so we can decom emery
|
|
2014-04-08 16:24:59
|
<ottomata>
|
but it couldn't talk to statsd
|
|
2014-04-08 16:25:38
|
<ottomata>
|
hm.
|
|
2014-04-08 16:25:44
|
<ottomata>
|
yeah totally working now
|
|
2014-04-08 16:25:57
|
<ottomata>
|
ooooook.
|
|
2014-04-08 16:25:59
|
<ottomata>
|
weird.
|
|
2014-04-08 16:26:00
|
<_joe_>
|
ottomata: graphite is borked
|
|
2014-04-08 16:26:04
|
<mark>
|
i think faidon did it earlier
|
|
2014-04-08 16:26:05
|
<grrrit-wm>
|
('CR') 'Hashar': "puppet is broken on deployment-bastion.eqiad.wmflabs, can't deploy the change right now :-/" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124624' (owner: 'Hashar')
|
|
2014-04-08 16:26:21
|
<ottomata>
|
oh, fixed the acl problem?
|
|
2014-04-08 16:26:33
|
<ottomata>
|
maybe something else was just not working, and I assumed because I couldn't ping it was an ACL thing?
|
|
2014-04-08 16:26:55
|
<mark>
|
ping is not a good way to test that
|
|
2014-04-08 16:27:10
|
<ottomata>
|
yeah, i just saw the packets being filtered from ping
|
|
2014-04-08 16:27:11
|
<mark>
|
we allow specific protocols/ports, ping uses different ones
|
|
2014-04-08 16:27:14
|
<ottomata>
|
aye
|
|
2014-04-08 16:27:30
|
<ottomata>
|
yeah, just figured if i couldn't at least ping then probably other stuff was blcoked too, but ja
|
|
2014-04-08 16:27:57
|
<ottomata>
|
but yeah, ori couldn't use apt on vanadium either, so dunno...
|
|
2014-04-08 16:28:10
|
<ottomata>
|
and sqstat couldnt' talk to tungsten, so hm
|
|
2014-04-08 16:28:12
|
<ottomata>
|
but ok!
|
|
2014-04-08 16:28:16
|
<mark>
|
:)
|
|
2014-04-08 16:28:22
|
<mark>
|
we're going for dinner in a bit
|
|
2014-04-08 16:28:44
|
<ottomata>
|
mark
|
|
2014-04-08 16:28:45
|
<ottomata>
|
hm
|
|
2014-04-08 16:28:53
|
<ottomata>
|
so sqstat is trying to talk to tungsten on 2003
|
|
2014-04-08 16:28:56
|
<hashar>
|
!log Jenkins: killed jenkins-slave java process on gallium and repooled gallium slave. It was no more registered in Zuul :-/
|
|
2014-04-08 16:28:57
|
<icinga-wm>
|
RECOVERY - puppet disabled on iron is OK: OK
|
|
2014-04-08 16:28:57
|
<ottomata>
|
is that open?
|
|
2014-04-08 16:29:01
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 16:29:07
|
<icinga-wm>
|
RECOVERY - Disk space on iron is OK: DISK OK
|
|
2014-04-08 16:29:09
|
<ottomata>
|
can't seem to reach it from an03
|
|
2014-04-08 16:29:34
|
<manybubbles>
|
ganglia seems upset
|
|
2014-04-08 16:29:40
|
<mark>
|
protocol udp;
|
|
2014-04-08 16:29:40
|
<mark>
|
destination-port 8125;
|
|
2014-04-08 16:29:45
|
<aude>
|
tables added
|
|
2014-04-08 16:29:51
|
<mark>
|
so port 2003 isn't
|
|
2014-04-08 16:29:54
|
<ottomata>
|
ah ok
|
|
2014-04-08 16:30:03
|
<ottomata>
|
that's why then, could you add?
|
|
2014-04-08 16:30:13
|
<mark>
|
ok
|
|
2014-04-08 16:30:40
|
<ottomata>
|
i'm going to see if reqstats gets flaky when we move it to analytics1003
|
|
2014-04-08 16:30:51
|
<ottomata>
|
it was either flaky because erbium is busy
|
|
2014-04-08 16:30:57
|
<ottomata>
|
or because the multicast firehose is just too lossy
|
|
2014-04-08 16:31:37
|
<aude>
|
!log added sites and site_identifiers core tables on wikiquote
|
|
2014-04-08 16:31:41
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 16:32:22
|
<mark>
|
2003 should work now
|
|
2014-04-08 16:33:36
|
<icinga-wm>
|
RECOVERY - DPKG on iodine is OK: All packages OK
|
|
2014-04-08 16:33:36
|
<icinga-wm>
|
RECOVERY - Disk space on iodine is OK: DISK OK
|
|
2014-04-08 16:33:36
|
<icinga-wm>
|
RECOVERY - puppet disabled on cp3003 is OK: OK
|
|
2014-04-08 16:33:39
|
<ottomata>
|
ah just noticed it is udp, mark, will that work still?
|
|
2014-04-08 16:33:46
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4014 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:46
|
<icinga-wm>
|
RECOVERY - RAID on cp3003 is OK: OK: optimal, 2 logical, 2 physical
|
|
2014-04-08 16:33:46
|
<icinga-wm>
|
RECOVERY - RAID on iodine is OK: OK: no disks configured for RAID
|
|
2014-04-08 16:33:46
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1005 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:46
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4003 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:47
|
<mark>
|
yes
|
|
2014-04-08 16:33:51
|
<ottomata>
|
ok cool
|
|
2014-04-08 16:33:52
|
<ottomata>
|
thanks
|
|
2014-04-08 16:33:53
|
<ottomata>
|
ok go eat
|
|
2014-04-08 16:33:55
|
<ottomata>
|
thank you!
|
|
2014-04-08 16:33:56
|
<icinga-wm>
|
RECOVERY - DPKG on bast1001 is OK: All packages OK
|
|
2014-04-08 16:33:56
|
<icinga-wm>
|
RECOVERY - puppet disabled on iodine is OK: OK
|
|
2014-04-08 16:33:56
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4002 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:56
|
<icinga-wm>
|
RECOVERY - HTTPS on amssq47 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:56
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4004 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:57
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4001 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:57
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4017 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:58
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4015 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:58
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4008 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:59
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1001 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:33:59
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4005 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:34:00
|
<icinga-wm>
|
RECOVERY - Disk space on bast1001 is OK: DISK OK
|
|
2014-04-08 16:34:00
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4019 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:34:06
|
<icinga-wm>
|
RECOVERY - RAID on bast1001 is OK: OK: no RAID installed
|
|
2014-04-08 16:34:06
|
<icinga-wm>
|
RECOVERY - DPKG on cp3003 is OK: All packages OK
|
|
2014-04-08 16:34:06
|
<icinga-wm>
|
RECOVERY - HTTPS on ssl1003 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:34:06
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4007 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:34:16
|
<icinga-wm>
|
RECOVERY - puppet disabled on bast1001 is OK: OK
|
|
2014-04-08 16:34:16
|
<icinga-wm>
|
RECOVERY - Disk space on cp3003 is OK: DISK OK
|
|
2014-04-08 16:34:16
|
<icinga-wm>
|
RECOVERY - HTTPS on cp4011 is OK: SSL_CERT OK - X.509 certificate for *.wikipedia.org from DigiCert High Assurance CA-3 valid until Jan 20 12:00:00 2016 GMT (expires in 652 days)
|
|
2014-04-08 16:35:36
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs4004 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 01:35:09 PM UTC
|
|
2014-04-08 16:35:46
|
<icinga-wm>
|
PROBLEM - HTTPS on cp1044 is CRITICAL: SSL_CERT CRITICAL *.wikimedia.org: invalid CN (*.wikimedia.org does not match *.wikipedia.org)
|
|
2014-04-08 16:35:56
|
<icinga-wm>
|
PROBLEM - HTTPS on cp1043 is CRITICAL: SSL_CERT CRITICAL *.wikimedia.org: invalid CN (*.wikimedia.org does not match *.wikipedia.org)
|
|
2014-04-08 16:36:48
|
<grrrit-wm>
|
('PS1') 'Ottomata': Putting sqstat back on analytics1003 [operations/puppet] - 'https://gerrit.wikimedia.org/r/124630'
|
|
2014-04-08 16:37:16
|
<grrrit-wm>
|
('CR') 'Ottomata': [C: '2' V: '2'] Putting sqstat back on analytics1003 [operations/puppet] - 'https://gerrit.wikimedia.org/r/124630' (owner: 'Ottomata')
|
|
2014-04-08 16:38:30
|
<grrrit-wm>
|
('PS1') 'Springle': invalid MariaDB variable name: user_stat [operations/puppet] - 'https://gerrit.wikimedia.org/r/124632'
|
|
2014-04-08 16:40:40
|
<grrrit-wm>
|
('CR') 'Springle': [C: '2'] invalid MariaDB variable name: user_stat [operations/puppet] - 'https://gerrit.wikimedia.org/r/124632' (owner: 'Springle')
|
|
2014-04-08 16:46:50
|
<grrrit-wm>
|
('PS1') 'RobH': replace misc-web-lb cert [operations/puppet] - 'https://gerrit.wikimedia.org/r/124634'
|
|
2014-04-08 16:48:11
|
<grrrit-wm>
|
('CR') 'RobH': [C: '2' V: '2'] replace misc-web-lb cert [operations/puppet] - 'https://gerrit.wikimedia.org/r/124634' (owner: 'RobH')
|
|
2014-04-08 16:49:09
|
<aude>
|
sorry, being slow... populating sites table
|
|
2014-04-08 16:49:20
|
<grrrit-wm>
|
('PS1') 'Alexandros Kosiaris': Removing ethtool package from other places [operations/puppet] - 'https://gerrit.wikimedia.org/r/124637'
|
|
2014-04-08 16:49:22
|
<aude>
|
suppose no hurry
|
|
2014-04-08 16:50:08
|
<grrrit-wm>
|
('CR') 'Dzahn': [C: ''] Removing ethtool package from other places [operations/puppet] - 'https://gerrit.wikimedia.org/r/124637' (owner: 'Alexandros Kosiaris')
|
|
2014-04-08 16:52:03
|
<grrrit-wm>
|
('CR') 'Dzahn': [C: '2'] "now included in base" [operations/puppet] - 'https://gerrit.wikimedia.org/r/124637' (owner: 'Alexandros Kosiaris')
|
|
2014-04-08 16:53:08
|
<grrrit-wm>
|
('CR') 'Cmcmahon': [C: ''] "Thanks for putting this back." [operations/puppet] - 'https://gerrit.wikimedia.org/r/124624' (owner: 'Hashar')
|
|
2014-04-08 16:53:36
|
<icinga-wm>
|
RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Tue Apr 8 16:53:29 UTC 2014
|
|
2014-04-08 16:53:46
|
<icinga-wm>
|
RECOVERY - Puppet freshness on dataset1001 is OK: puppet ran at Tue Apr 8 16:53:39 UTC 2014
|
|
2014-04-08 16:55:06
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 16:55:28
|
<ottomata>
|
rats
|
|
2014-04-08 16:56:36
|
<icinga-wm>
|
RECOVERY - Puppet freshness on amslvs2 is OK: puppet ran at Tue Apr 8 16:56:30 UTC 2014
|
|
2014-04-08 16:56:46
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs1003 is OK: puppet ran at Tue Apr 8 16:56:45 UTC 2014
|
|
2014-04-08 16:59:04
|
<aude>
|
waiting for jenkins
|
|
2014-04-08 17:01:46
|
<icinga-wm>
|
RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Tue Apr 8 17:01:37 UTC 2014
|
|
2014-04-08 17:01:48
|
<grrrit-wm>
|
('PS2') 'Manybubbles': Turn on experimental highlighting in beta [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124003'
|
|
2014-04-08 17:03:06
|
<logmsgbot>
|
!log aude synchronized php-1.23wmf20/extensions/Wikidata 'Update Wikidata build, to allow populating sites table on wikiquote'
|
|
2014-04-08 17:03:10
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:05:20
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs4004 is OK: puppet ran at Tue Apr 8 17:05:14 UTC 2014
|
|
2014-04-08 17:05:30
|
<icinga-wm>
|
PROBLEM - RAID on dataset1001 is CRITICAL: CRITICAL: 1 failed LD(s) (Partially Degraded)
|
|
2014-04-08 17:06:40
|
<icinga-wm>
|
PROBLEM - LVS HTTPS IPv6 on misc-web-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused
|
|
2014-04-08 17:07:40
|
<icinga-wm>
|
RECOVERY - LVS HTTPS IPv6 on misc-web-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 226 bytes in 0.012 second response time
|
|
2014-04-08 17:08:20
|
<icinga-wm>
|
RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Tue Apr 8 17:08:15 UTC 2014
|
|
2014-04-08 17:08:30
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs4003 is OK: puppet ran at Tue Apr 8 17:08:25 UTC 2014
|
|
2014-04-08 17:08:44
|
<grrrit-wm>
|
('CR') 'Chad': [C: '2'] Turn on experimental highlighting in beta [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124003' (owner: 'Manybubbles')
|
|
2014-04-08 17:08:53
|
<grrrit-wm>
|
('Merged') 'jenkins-bot': Turn on experimental highlighting in beta [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124003' (owner: 'Manybubbles')
|
|
2014-04-08 17:09:40
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs1006 is OK: puppet ran at Tue Apr 8 17:09:30 UTC 2014
|
|
2014-04-08 17:10:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 17:10:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 17:10:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 17:10:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 17:10:19
|
<grrrit-wm>
|
('CR') 'QChris': "Prerequisite got merged." [operations/puppet] - 'https://gerrit.wikimedia.org/r/121546' (owner: 'Ottomata')
|
|
2014-04-08 17:10:52
|
<aude>
|
^demon|away: are you deploying stuff?
|
|
2014-04-08 17:11:14
|
<aude>
|
i'll need to sneak in some point for a config change, but not yet
|
|
2014-04-08 17:11:29
|
<grrrit-wm>
|
('PS1') 'Ottomata': Moving sqstat back to emery :/ [operations/puppet] - 'https://gerrit.wikimedia.org/r/124641'
|
|
2014-04-08 17:11:38
|
<grrrit-wm>
|
('PS2') 'Ottomata': Moving sqstat back to emery :/ [operations/puppet] - 'https://gerrit.wikimedia.org/r/124641'
|
|
2014-04-08 17:11:40
|
<grrrit-wm>
|
('CR') 'jenkins-bot': [V: '-1'] Moving sqstat back to emery :/ [operations/puppet] - 'https://gerrit.wikimedia.org/r/124641' (owner: 'Ottomata')
|
|
2014-04-08 17:11:50
|
<grrrit-wm>
|
('CR') 'Ottomata': [C: '2' V: '2'] Moving sqstat back to emery :/ [operations/puppet] - 'https://gerrit.wikimedia.org/r/124641' (owner: 'Ottomata')
|
|
2014-04-08 17:12:28
|
<manybubbles>
|
aude: no, he just merged something for beta
|
|
2014-04-08 17:12:34
|
<aude>
|
ok
|
|
2014-04-08 17:12:41
|
<aude>
|
probably need 10 more minutes
|
|
2014-04-08 17:12:50
|
<aude>
|
done populating tables, now checking they are ok
|
|
2014-04-08 17:13:00
|
<aude>
|
then can do the config change and then done :)
|
|
2014-04-08 17:13:19
|
<^demon|away>
|
aude: Nope, just merged that for Nik for beta.
|
|
2014-04-08 17:13:21
|
<^demon|away>
|
Like he said :)
|
|
2014-04-08 17:13:22
|
<aude>
|
going slow and careful since i'm still newish
|
|
2014-04-08 17:13:25
|
<aude>
|
doign this stuff
|
|
2014-04-08 17:13:32
|
<^demon|away>
|
Someone should sync it eventually for consistency, but no biggie.
|
|
2014-04-08 17:13:53
|
<aude>
|
i can do
|
|
2014-04-08 17:14:04
|
<hoo>
|
so can I
|
|
2014-04-08 17:14:29
|
<aude>
|
hoo: want to check the sites tables and site_identifiers for wikiquote?
|
|
2014-04-08 17:14:30
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs1002 is OK: puppet ran at Tue Apr 8 17:14:22 UTC 2014
|
|
2014-04-08 17:14:36
|
<aude>
|
they look ok to me
|
|
2014-04-08 17:15:30
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs1005 is OK: puppet ran at Tue Apr 8 17:15:22 UTC 2014
|
|
2014-04-08 17:16:02
|
<grrrit-wm>
|
('CR') 'Aude': "sites table and site_identifiers are added and populated" [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124516' (owner: 'Aude')
|
|
2014-04-08 17:16:10
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs1004 is OK: puppet ran at Tue Apr 8 17:16:02 UTC 2014
|
|
2014-04-08 17:16:28
|
<manybubbles>
|
!log finished upgrading elastic1001-1006. starting on 1007. yay progress.
|
|
2014-04-08 17:16:32
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:16:34
|
<hoo>
|
enwikiqoute looks good to me
|
|
2014-04-08 17:16:39
|
<aude>
|
alright
|
|
2014-04-08 17:16:40
|
<hoo>
|
sites and site_identifiers
|
|
2014-04-08 17:16:44
|
<aude>
|
strip protocals and all
|
|
2014-04-08 17:16:52
|
<hoo>
|
yep
|
|
2014-04-08 17:16:58
|
<aude>
|
https://gerrit.wikimedia.org/r/#/c/124516/ want to merge
|
|
2014-04-08 17:17:07
|
<aude>
|
i can deploy it and sync the cirrus thing
|
|
2014-04-08 17:17:19
|
<manybubbles>
|
thanks1
|
|
2014-04-08 17:17:22
|
<hoo>
|
ok, also looks good on WD
|
|
2014-04-08 17:17:30
|
<aude>
|
ok
|
|
2014-04-08 17:17:45
|
<aude>
|
let me sync cirrus
|
|
2014-04-08 17:17:52
|
<hoo>
|
go ahead
|
|
2014-04-08 17:17:53
|
<Nemo_bis>
|
Oh, today is the day
|
|
2014-04-08 17:18:06
|
<aude>
|
it's *the* day :)
|
|
2014-04-08 17:18:10
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs4001 is OK: puppet ran at Tue Apr 8 17:18:03 UTC 2014
|
|
2014-04-08 17:19:18
|
<hoo>
|
aude: You also sorted the wikidataclient dblist? :P
|
|
2014-04-08 17:19:53
|
<aude>
|
yes
|
|
2014-04-08 17:20:04
|
<hoo>
|
Ok, looks good to me, can approve whenever you want
|
|
2014-04-08 17:20:05
|
<aude>
|
they will get sorted eventually
|
|
2014-04-08 17:20:13
|
<aude>
|
doing chad's thing
|
|
2014-04-08 17:20:30
|
<icinga-wm>
|
RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Tue Apr 8 17:20:23 UTC 2014
|
|
2014-04-08 17:21:30
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs1001 is OK: puppet ran at Tue Apr 8 17:21:24 UTC 2014
|
|
2014-04-08 17:21:50
|
<icinga-wm>
|
RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Tue Apr 8 17:21:45 UTC 2014
|
|
2014-04-08 17:22:30
|
<icinga-wm>
|
RECOVERY - Puppet freshness on lvs4002 is OK: puppet ran at Tue Apr 8 17:22:21 UTC 2014
|
|
2014-04-08 17:22:43
|
<logmsgbot>
|
!log aude synchronized wmf-config/CirrusSearch-labs.php 'config change for beta, to enable highlighting'
|
|
2014-04-08 17:22:47
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:23:06
|
<aude>
|
hoo: ready
|
|
2014-04-08 17:23:45
|
<grrrit-wm>
|
('CR') 'Hoo man': [C: '2'] "Preparation finished, so do this! \o/" [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124516' (owner: 'Aude')
|
|
2014-04-08 17:23:49
|
<aude>
|
yay!
|
|
2014-04-08 17:23:51
|
<hoo>
|
there you go ;)
|
|
2014-04-08 17:23:53
|
<grrrit-wm>
|
('Merged') 'jenkins-bot': Enable Wikibase on Wikiquote [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124516' (owner: 'Aude')
|
|
2014-04-08 17:27:20
|
<hoo>
|
aude: About to sync or shall I take it?
|
|
2014-04-08 17:27:21
|
<aude>
|
sync dblist then wmf-config?
|
|
2014-04-08 17:27:31
|
<Nemo_bis>
|
waiting
|
|
2014-04-08 17:27:43
|
<aude>
|
no other way
|
|
2014-04-08 17:27:52
|
<hoo>
|
other way round sounds sane
|
|
2014-04-08 17:28:02
|
<aude>
|
wmf-config then dblist is good
|
|
2014-04-08 17:28:06
|
<hoo>
|
wmf-config changes will work w/o the rest
|
|
2014-04-08 17:28:10
|
<aude>
|
right
|
|
2014-04-08 17:28:20
|
<aude>
|
that' what ree-dy did for wikisource
|
|
2014-04-08 17:28:52
|
<aude>
|
doing
|
|
2014-04-08 17:28:55
|
<hoo>
|
:)
|
|
2014-04-08 17:28:59
|
<logmsgbot>
|
!log aude synchronized wmf-config 'config changes to enable Wikibase on Wikiquote'
|
|
2014-04-08 17:29:04
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:29:12
|
<grrrit-wm>
|
('PS1') 'Matthias Mullie': Increase Flow cache version [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124646'
|
|
2014-04-08 17:29:52
|
<logmsgbot>
|
!log aude synchronized wikidataclient.dblist 'Enable Wikibase on Wikiquote'
|
|
2014-04-08 17:29:57
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:30:01
|
<hoo>
|
oO
|
|
2014-04-08 17:30:02
|
<hoo>
|
:)
|
|
2014-04-08 17:30:12
|
<aude>
|
alright time to check it's all good
|
|
2014-04-08 17:30:17
|
<hoo>
|
on that
|
|
2014-04-08 17:31:13
|
<hoo>
|
oh well... I think we have to bump wgCacheEpoch once again
|
|
2014-04-08 17:31:14
|
<hoo>
|
aude: ^
|
|
2014-04-08 17:31:36
|
<aude>
|
huh
|
|
2014-04-08 17:31:45
|
<aude>
|
ah, yes
|
|
2014-04-08 17:32:00
|
<hoo>
|
shall I patch or will you?
|
|
2014-04-08 17:32:26
|
<Nemo_bis>
|
https://www.wikidata.org/wiki/Q189119#sitelinks-wikiquote
|
|
2014-04-08 17:32:34
|
<hoo>
|
Nemo_bis: Yes, the usual stuff
|
|
2014-04-08 17:32:34
|
<aude>
|
go ahead
|
|
2014-04-08 17:33:06
|
<aude>
|
it says list of values is complete
|
|
2014-04-08 17:33:09
|
<aude>
|
i assume caching
|
|
2014-04-08 17:33:16
|
<aude>
|
on Q60
|
|
2014-04-08 17:33:57
|
<aude>
|
debug=true, i can add wikiquote
|
|
2014-04-08 17:34:23
|
<Nemo_bis>
|
yep, I did action=purge
|
|
2014-04-08 17:34:23
|
<grrrit-wm>
|
('PS1') 'Hoo man': Bump wgCacheEpoch for Wikidata after enabling Wikiquote langlinks [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124648'
|
|
2014-04-08 17:34:24
|
<hoo>
|
yep
|
|
2014-04-08 17:34:31
|
<hoo>
|
aude: ^
|
|
2014-04-08 17:34:35
|
<aude>
|
ok
|
|
2014-04-08 17:35:21
|
<ottomata>
|
!log restarted gmetad on nickel to fix ganglia
|
|
2014-04-08 17:35:26
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:35:33
|
<grrrit-wm>
|
('CR') 'Aude': [C: '2'] Bump wgCacheEpoch for Wikidata after enabling Wikiquote langlinks [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124648' (owner: 'Hoo man')
|
|
2014-04-08 17:35:40
|
<grrrit-wm>
|
('Merged') 'jenkins-bot': Bump wgCacheEpoch for Wikidata after enabling Wikiquote langlinks [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124648' (owner: 'Hoo man')
|
|
2014-04-08 17:37:00
|
<hoo>
|
aude: Syncing? I have to sync a touch out
|
|
2014-04-08 17:37:10
|
<aude>
|
doing
|
|
2014-04-08 17:37:12
|
<hoo>
|
ok
|
|
2014-04-08 17:37:18
|
<logmsgbot>
|
!log aude synchronized wmf-config/Wikibase.php 'bump wgCacheEpoch for wikidata after enabling wikiquote site links'
|
|
2014-04-08 17:37:19
|
<aude>
|
just being careful
|
|
2014-04-08 17:37:22
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:37:28
|
<logmsgbot>
|
!log hoo synchronized php-1.23wmf20/extensions/Wikidata/extensions/Wikibase/lib/resources/wikibase.Site.js 'touch'
|
|
2014-04-08 17:37:32
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:37:34
|
<hoo>
|
that should purge the sites cache
|
|
2014-04-08 17:37:43
|
<greg-g>
|
"13:37 < aude> just being careful" +1 ;)
|
|
2014-04-08 17:37:44
|
<hoo>
|
in resource loader
|
|
2014-04-08 17:37:47
|
<aude>
|
:)
|
|
2014-04-08 17:38:25
|
<aude>
|
still says complete
|
|
2014-04-08 17:38:30
|
<hoo>
|
mh :/
|
|
2014-04-08 17:38:45
|
<aude>
|
sites module has always been a pain
|
|
2014-04-08 17:40:24
|
<aude>
|
maybe php-1.23wmf20/extensions/Wikidata/extensions/Wikibase/lib/includes/modules/SitesModule.php ?
|
|
2014-04-08 17:40:43
|
<hoo>
|
aude: Wont help, RL does timestamps based on the JS scripts
|
|
2014-04-08 17:40:50
|
<aude>
|
hmmm, ok
|
|
2014-04-08 17:41:13
|
<hoo>
|
works for me
|
|
2014-04-08 17:41:16
|
<hoo>
|
now at least
|
|
2014-04-08 17:41:35
|
<aude>
|
trying in firefox
|
|
2014-04-08 17:41:39
|
<aude>
|
might be my caching
|
|
2014-04-08 17:41:42
|
<hoo>
|
\o/ Just added the first link
|
|
2014-04-08 17:41:46
|
<hoo>
|
https://www.wikidata.org/wiki/Q40904#sitelinks-wikiquote
|
|
2014-04-08 17:41:48
|
<aude>
|
already did one :)
|
|
2014-04-08 17:41:54
|
<aude>
|
with debug=true
|
|
2014-04-08 17:41:59
|
<hoo>
|
Cheating :D
|
|
2014-04-08 17:42:11
|
<aude>
|
heh
|
|
2014-04-08 17:42:23
|
<aude>
|
looks good in firefox
|
|
2014-04-08 17:42:30
|
<aude>
|
i have to assume it's my cache
|
|
2014-04-08 17:42:31
|
<Nemo_bis>
|
I did one ten minutes ago already :P
|
|
2014-04-08 17:42:35
|
<hoo>
|
:P
|
|
2014-04-08 17:42:36
|
<aude>
|
yay
|
|
2014-04-08 17:42:45
|
<hoo>
|
Nemo_bis: with debug true, I guess?!
|
|
2014-04-08 17:42:50
|
<Nemo_bis>
|
lol Heisenberg
|
|
2014-04-08 17:42:55
|
<Nemo_bis>
|
19.34 < Nemo_bis> yep, I did action=purge
|
|
2014-04-08 17:43:01
|
<hoo>
|
:P
|
|
2014-04-08 17:43:01
|
<aude>
|
ah
|
|
2014-04-08 17:43:50
|
<Guest75555>
|
Is there a procedure to delete gerrit repositories?
|
|
2014-04-08 17:45:00
|
<aude>
|
i can add links in wikidata now in chrome
|
|
2014-04-08 17:45:09
|
<hoo>
|
aude: https://en.wikiquote.org/w/index.php?title=Werner_Heisenberg&action=info mh
|
|
2014-04-08 17:45:14
|
<hoo>
|
why is it not showing up?
|
|
2014-04-08 17:45:34
|
<Nemo_bis>
|
Guest64226 / krinkle : probably you can ask on the same gerrit queue page as usual
|
|
2014-04-08 17:45:53
|
<hoo>
|
ah, I see
|
|
2014-04-08 17:45:57
|
<Nemo_bis>
|
unless it's not "your" repository, in which case maybe a bug is better
|
|
2014-04-08 17:46:11
|
<hoo>
|
dispatching is ... :S
|
|
2014-04-08 17:47:21
|
<aude>
|
hmmm
|
|
2014-04-08 17:47:28
|
<hoo>
|
https://www.wikidata.org/wiki/Special:DispatchStats
|
|
2014-04-08 17:47:44
|
<aude>
|
i did action=purge on https://en.wikiquote.org/wiki/New_York_City
|
|
2014-04-08 17:47:46
|
<hoo>
|
aude: Can we safely skip theses changes? If not just waiting is also fine
|
|
2014-04-08 17:47:54
|
<hoo>
|
it's catching up rather quickly AFAIS
|
|
2014-04-08 17:47:55
|
<aude>
|
removed dewikiquote
|
|
2014-04-08 17:48:08
|
<aude>
|
we can wait
|
|
2014-04-08 17:48:16
|
<bd808|deploy>
|
waits in line to do a group0 to 1.23wmf21 scap
|
|
2014-04-08 17:48:28
|
<aude>
|
give us 5 more minutes to poke
|
|
2014-04-08 17:48:43
|
<bd808|deploy>
|
aude: Sounds good
|
|
2014-04-08 17:48:59
|
<aude>
|
i think we're ok though...
|
|
2014-04-08 17:49:32
|
<aude>
|
or nothing we solve in 5 min, but didn't break anything
|
|
2014-04-08 17:50:51
|
<hoo>
|
aude: I can bump the chd_seen fields
|
|
2014-04-08 17:51:12
|
<aude>
|
ok
|
|
2014-04-08 17:52:05
|
<hoo>
|
Just looking for the right change id
|
|
2014-04-08 17:53:43
|
<hoo>
|
got that
|
|
2014-04-08 17:54:37
|
<aude>
|
something is weird with wikiquote... like it's not actually enabled now
|
|
2014-04-08 17:54:45
|
<aude>
|
but sure i saw it was
|
|
2014-04-08 17:55:29
|
<aude>
|
thinks this happened with wikisource
|
|
2014-04-08 17:56:19
|
<hoo>
|
!log changed the Wikidata wb_changes_dispatch position of all wikiquote wikis to 118158153
|
|
2014-04-08 17:56:23
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 17:56:39
|
<aude>
|
enwikiquote is in wikidataclient.dblist
|
|
2014-04-08 17:56:42
|
<hoo>
|
20140408172900
|
|
2014-04-08 17:57:03
|
<hoo>
|
that was the timestamp, should be a few moments before anything happened regarding wikiquote
|
|
2014-04-08 17:57:12
|
<aude>
|
ok
|
|
2014-04-08 17:57:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 540.333313
|
|
2014-04-08 17:58:28
|
<hoo>
|
still https://en.wikiquote.org/w/index.php?title=Werner_Heisenberg&action=info
|
|
2014-04-08 17:58:56
|
<hoo>
|
Wikidata is not even loaded there... wtf
|
|
2014-04-08 17:58:59
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 645.666687
|
|
2014-04-08 17:59:03
|
<aude>
|
right,
|
|
2014-04-08 17:59:05
|
<aude>
|
i'm sure it was
|
|
2014-04-08 17:59:25
|
<aude>
|
do i have to sync dblist again?
|
|
2014-04-08 17:59:37
|
<aude>
|
did we somehow undo it?
|
|
2014-04-08 18:00:58
|
<hoo>
|
no, looks good on a random mw* machine
|
|
2014-04-08 18:01:09
|
<icinga-wm>
|
PROBLEM - Disk space on virt1000 is CRITICAL: DISK CRITICAL - free space: / 1694 MB (2% inode=86%):
|
|
2014-04-08 18:01:14
|
<hoo>
|
ah
|
|
2014-04-08 18:01:50
|
<logmsgbot>
|
!log hoo synchronized wmf-config/InitialiseSettings.php 'Touch to clear config. cache'
|
|
2014-04-08 18:01:54
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 18:01:55
|
<aude>
|
ok
|
|
2014-04-08 18:02:09
|
<aude>
|
it's back!
|
|
2014-04-08 18:02:11
|
<hoo>
|
Sorry, I forgot about that
|
|
2014-04-08 18:02:33
|
<aude>
|
was about to try that
|
|
2014-04-08 18:02:37
|
<hoo>
|
:)
|
|
2014-04-08 18:02:41
|
<aude>
|
touch all the wikidata things :)
|
|
2014-04-08 18:02:43
|
<bd808|deploy>
|
wants to fix https://bugzilla.wikimedia.org/show_bug.cgi?id=58618 so that's automatic
|
|
2014-04-08 18:02:56
|
<aude>
|
i think we are done!
|
|
2014-04-08 18:03:19
|
<aude>
|
i am sure this happened on wikisource or previously where it was enabled and then not
|
|
2014-04-08 18:03:38
|
<aude>
|
puzzled but we're good now
|
|
2014-04-08 18:04:13
|
<hoo>
|
Yep, looks good to me
|
|
2014-04-08 18:04:23
|
<bd808|deploy>
|
aude, hoo: All clear for me to mess with /a/common on tin and then scap?
|
|
2014-04-08 18:04:37
|
<hoo>
|
Yep, go ahead... we're done for now :)
|
|
2014-04-08 18:04:47
|
<bd808|deploy>
|
Cool
|
|
2014-04-08 18:05:08
|
<aude>
|
done
|
|
2014-04-08 18:06:11
|
<grrrit-wm>
|
('PS1') 'BryanDavis': Group0 wikis to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124655'
|
|
2014-04-08 18:06:50
|
<greg-g>
|
crosses fingers and knocks on wood
|
|
2014-04-08 18:07:03
|
<grrrit-wm>
|
('CR') 'BryanDavis': [C: '2'] Group0 wikis to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124655' (owner: 'BryanDavis')
|
|
2014-04-08 18:07:05
|
<aude>
|
too!
|
|
2014-04-08 18:07:46
|
<bd808|deploy>
|
greg-g: Aaron merged my fix so in theory I should only need one scap. I'll verify the file after the first scap to be certain
|
|
2014-04-08 18:08:21
|
<greg-g>
|
nods
|
|
2014-04-08 18:08:28
|
<grrrit-wm>
|
('Merged') 'jenkins-bot': Group0 wikis to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124655' (owner: 'BryanDavis')
|
|
2014-04-08 18:10:36
|
<logmsgbot>
|
!log bd808 Started scap: group0 wikis to 1.23wmf21 (with patch for bug 63659)
|
|
2014-04-08 18:10:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 18:10:41
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 18:11:25
|
<bd808|deploy>
|
l10n cache did not rebuild which is a great sign
|
|
2014-04-08 18:11:58
|
<jackmcbarn>
|
Unable to open /usr/local/apache/common-local/wikiversions.cdb.
|
|
2014-04-08 18:11:58
|
<MatmaRex>
|
https://pl.wikipedia.org/w/index.php?title=Dyskusja_wikiprojektu:%C5%9Ar%C3%B3dziemie&oldid=prev&diff=39218000
|
|
2014-04-08 18:12:01
|
<MatmaRex>
|
i get a "Unable to open /usr/local/apache/common-local/wikiversions.cdb."
|
|
2014-04-08 18:12:10
|
<andre__>
|
...and same here.
|
|
2014-04-08 18:12:12
|
<manybubbles>
|
[2014-04-08 18:11:37] Fatal error: Unable to open /usr/local/apache/common-local/wikiversions.cdb.
|
|
2014-04-08 18:12:15
|
<rschen7754>
|
uh-oh
|
|
2014-04-08 18:12:19
|
<bd808|deploy>
|
Yeah. fuck
|
|
2014-04-08 18:12:21
|
<manybubbles>
|
yeah, you got it
|
|
2014-04-08 18:12:22
|
<Steinsplitter>
|
here the same
|
|
2014-04-08 18:12:26
|
<bd808|deploy>
|
It will be fixed in a few moments
|
|
2014-04-08 18:12:30
|
<manybubbles>
|
thats everything
|
|
2014-04-08 18:12:31
|
<greg-g>
|
well shit
|
|
2014-04-08 18:12:45
|
<bd808|deploy>
|
fuuuuck
|
|
2014-04-08 18:12:49
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000
|
|
2014-04-08 18:12:57
|
<bd808|deploy>
|
There's my first crash all of the wikis
|
|
2014-04-08 18:12:59
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 18:13:00
|
<MaxSem>
|
SNAFU?
|
|
2014-04-08 18:13:05
|
<aude>
|
wtf
|
|
2014-04-08 18:13:13
|
<Amgine>
|
down on wm
|
|
2014-04-08 18:13:21
|
<manybubbles>
|
damn it, I was actually reading an article and I reloaded it to test
|
|
2014-04-08 18:13:23
|
<bd808|deploy>
|
It was my "fix" for the scap problem
|
|
2014-04-08 18:13:25
|
<manybubbles>
|
now I can't read it while I wait
|
|
2014-04-08 18:13:29
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1190 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.007 second response time
|
|
2014-04-08 18:13:29
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1055 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.013 second response time
|
|
2014-04-08 18:13:29
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1150 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.004 second response time
|
|
2014-04-08 18:13:29
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.005 second response time
|
|
2014-04-08 18:13:29
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1177 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.009 second response time
|
|
2014-04-08 18:13:29
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1138 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.003 second response time
|
|
2014-04-08 18:13:30
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1187 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.006 second response time
|
|
2014-04-08 18:13:30
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1220 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.006 second response time
|
|
2014-04-08 18:13:31
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1197 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.013 second response time
|
|
2014-04-08 18:13:31
|
<icinga-wm>
|
PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_job_queue) or PHP errors -
|
|
2014-04-08 18:13:33
|
<marktraceur>
|
Whoa
|
|
2014-04-08 18:13:34
|
<aude>
|
cries
|
|
2014-04-08 18:13:39
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1213 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.018 second response time
|
|
2014-04-08 18:13:39
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1113 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.012 second response time
|
|
2014-04-08 18:13:39
|
<icinga-wm>
|
PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.008 second response time
|
|
2014-04-08 18:13:42
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1200 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.006 second response time
|
|
2014-04-08 18:13:42
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1035 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.022 second response time
|
|
2014-04-08 18:13:42
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1031 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.011 second response time
|
|
2014-04-08 18:13:42
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1090 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.010 second response time
|
|
2014-04-08 18:13:42
|
<icinga-wm>
|
PROBLEM - Apache HTTP on mw1154 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal server error - 50485 bytes in 0.007 second response time
|
|
2014-04-08 18:13:52
|
<bd808|deploy>
|
It will be fixed soon… scap will fix it at the end
|
|
2014-04-08 18:13:54
|
<logmsgbot>
|
!log bd808 Finished scap: group0 wikis to 1.23wmf21 (with patch for bug 63659) (duration: 03m 18s)
|
|
2014-04-08 18:13:59
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 18:14:00
|
<aude>
|
alright
|
|
2014-04-08 18:14:01
|
<bd808|deploy>
|
Should be fixed now
|
|
2014-04-08 18:14:04
|
<manybubbles>
|
fixed
|
|
2014-04-08 18:14:15
|
<greg-g>
|
breathes again
|
|
2014-04-08 18:14:22
|
<jackmcbarn>
|
can whoever's in charge of icinga-wm bring it back to life?
|
|
2014-04-08 18:14:35
|
<sjoerddebruin>
|
Damn it. :P
|
|
2014-04-08 18:14:37
|
<greg-g>
|
jackmcbarn: it'll again automatically, I *believe*
|
|
2014-04-08 18:14:38
|
<PiRCarre>
|
Someone
|
|
2014-04-08 18:14:39
|
<MaxSem>
|
so what happened?
|
|
2014-04-08 18:14:47
|
<PiRCarre>
|
Oh, you know about it?
|
|
2014-04-08 18:14:48
|
<Marybelle>
|
greg-g: You accidentally a verb.
|
|
2014-04-08 18:14:49
|
<PiRCarre>
|
ok
|
|
2014-04-08 18:14:50
|
<icinga-wm>
|
RECOVERY - Apache HTTP on mw1027 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.066 second response time
|
|
2014-04-08 18:14:50
|
<icinga-wm>
|
RECOVERY - Apache HTTP on mw1092 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.073 second response time
|
|
2014-04-08 18:14:51
|
<icinga-wm>
|
RECOVERY - Apache HTTP on mw1073 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.084 second response time
|
|
2014-04-08 18:14:51
|
<icinga-wm>
|
RECOVERY - Apache HTTP on mw1018 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.111 second response time
|
|
2014-04-08 18:14:51
|
<bd808|deploy>
|
Patch https://gerrit.wikimedia.org/r/#/c/124627/
|
|
2014-04-08 18:14:52
|
<icinga-wm>
|
RECOVERY - Apache HTTP on mw1163 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.062 second response time
|
|
2014-04-08 18:14:52
|
<icinga-wm>
|
RECOVERY - Apache HTTP on mw1217 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.059 second response time
|
|
2014-04-08 18:15:07
|
<greg-g>
|
Marybelle: :)
|
|
2014-04-08 18:15:16
|
<bd808|deploy>
|
I'll write up the email. I know exactly what I fucked up
|
|
2014-04-08 18:15:21
|
<PiRCarre>
|
bd808|deploy: thanks, I was just about to report "Unable to open /usr/local/apache/common-local/wikiversions.cdb." - glad to see it's under control
|
|
2014-04-08 18:15:29
|
<aude>
|
breathes
|
|
2014-04-08 18:15:54
|
<paravoid>
|
what's going on?
|
|
2014-04-08 18:16:08
|
<paravoid>
|
we are all at dinner
|
|
2014-04-08 18:16:23
|
<manybubbles>
|
fixed now
|
|
2014-04-08 18:16:24
|
<aude>
|
it's ok
|
|
2014-04-08 18:16:25
|
<bd808|deploy>
|
paravoid: My fault. Should be fixed now
|
|
2014-04-08 18:16:31
|
<paravoid>
|
okay
|
|
2014-04-08 18:16:35
|
<greg-g>
|
paravoid: go back to dinner, all's ok again :)
|
|
2014-04-08 18:16:36
|
<aude>
|
scap temporarily broke everything though
|
|
2014-04-08 18:16:36
|
<paravoid>
|
do you need anything?
|
|
2014-04-08 18:16:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 183.266663
|
|
2014-04-08 18:16:39
|
<paravoid>
|
ok
|
|
2014-04-08 18:16:44
|
<paravoid>
|
manual page us if something happens
|
|
2014-04-08 18:16:52
|
<greg-g>
|
paravoid: nope, known ef up
|
|
2014-04-08 18:16:57
|
<greg-g>
|
paravoid: will do, enjoy!
|
|
2014-04-08 18:17:05
|
<paravoid>
|
ciao
|
|
2014-04-08 18:18:17
|
<grrrit-wm>
|
('PS2') 'Gerg? Tisza': Add setting to show a survey for MediaViewer users on some sites [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124036'
|
|
2014-04-08 18:18:56
|
<grrrit-wm>
|
('CR') 'Gerg? Tisza': "Updated to display feedback survey on beta enwiki." [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124036' (owner: 'Gerg? Tisza')
|
|
2014-04-08 18:19:29
|
<bd808|deploy>
|
greg-g: I just reverted my patch to scap that caused that cascade of horribleness
|
|
2014-04-08 18:19:36
|
<greg-g>
|
:)
|
|
2014-04-08 18:19:44
|
<bd808|deploy>
|
One the plus side, group0 is on wmf21 now
|
|
2014-04-08 18:19:50
|
<greg-g>
|
lol
|
|
2014-04-08 18:19:58
|
<greg-g>
|
literal-lol
|
|
2014-04-08 18:20:09
|
<aude>
|
scared to change it back
|
|
2014-04-08 18:20:20
|
<greg-g>
|
"Don't. Touch. Any. Thing."
|
|
2014-04-08 18:20:25
|
<aude>
|
i suppose if bd808|deploy 's patch is reverted then ok
|
|
2014-04-08 18:20:39
|
<greg-g>
|
well, we still have the previous issue which it was trying to fix ;)
|
|
2014-04-08 18:20:59
|
<greg-g>
|
1 step forward, 1 step back
|
|
2014-04-08 18:21:23
|
<bd808|deploy>
|
So yes we are temporarily back to needing to double-scap, but I'll make a patch that doesn't melt the world after lunch
|
|
2014-04-08 18:22:25
|
<greg-g>
|
bd808|deploy: :)
|
|
2014-04-08 18:23:15
|
<aude>
|
wikiquote etc all looks fine, so i'm going home / eating
|
|
2014-04-08 18:23:20
|
<aude>
|
back in hour
|
|
2014-04-08 18:23:26
|
<greg-g>
|
k, I'll do the same
|
|
2014-04-08 18:23:33
|
<Nemo_bis>
|
quite late dinner for berlin
|
|
2014-04-08 18:23:47
|
<manybubbles>
|
so I told my wife we broke the internet. she told me facebook was working....
|
|
2014-04-08 18:24:18
|
<hoo>
|
Nemo_bis: It's never to late for food :P
|
|
2014-04-08 18:24:41
|
<Jamesofur>
|
^
|
|
2014-04-08 18:28:38
|
<Nemo_bis>
|
hoo: well, I'd call death for starvation, pellagra etc. "too late" :P
|
|
2014-04-08 18:29:07
|
<hoo>
|
Nemo_bis: :P To late as in time of the day...
|
|
2014-04-08 18:29:08
|
<hoo>
|
:D
|
|
2014-04-08 18:30:17
|
<ori>
|
hoo: http://p.defau.lt/?md_cbLJuORDNsGkhY6_NAg :P
|
|
2014-04-08 18:30:55
|
<hoo>
|
at least the other errors are gone now, I guess
|
|
2014-04-08 18:31:28
|
<greg-g>
|
manybubbles: :(
|
|
2014-04-08 18:31:42
|
<greg-g>
|
goes to lunch for real
|
|
2014-04-08 18:32:34
|
<ori>
|
hoo: yeah, i submitted a patch for hhvm to fix that other issue btw
|
|
2014-04-08 18:32:49
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1012 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.144
|
|
2014-04-08 18:34:15
|
<hoo>
|
ori: Oh... nice that it's actually done in PHP :)
|
|
2014-04-08 18:35:34
|
<manybubbles>
|
yeah yeah yeah, elasticsearch 1012 is being upgraded
|
|
2014-04-08 18:37:56
|
<ori>
|
hoo: which component should that be filed under?
|
|
2014-04-08 18:39:25
|
<hoo>
|
ori: already done https://bugzilla.wikimedia.org/show_bug.cgi?id=63691
|
|
2014-04-08 18:39:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 639.299988
|
|
2014-04-08 18:39:40
|
<ori>
|
oh cool, thanks!
|
|
2014-04-08 18:42:09
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 530.733337
|
|
2014-04-08 18:42:20
|
<hoo>
|
ori: Any idea who to poke about https://gerrit.wikimedia.org/r/121709 ?
|
|
2014-04-08 18:43:46
|
<grrrit-wm>
|
('CR') 'Matanya': add interface speed check for all hosts ('2' comments) [operations/puppet] - 'https://gerrit.wikimedia.org/r/124606' (owner: 'Cmjohnson')
|
|
2014-04-08 18:44:08
|
<grrrit-wm>
|
('PS2') 'Ori.livneh': Change wgServer and wgCanonicalServer for arbcom wikis [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/121709' (owner: 'Hoo man')
|
|
2014-04-08 18:44:53
|
<grrrit-wm>
|
('CR') 'Ori.livneh': [C: '2'] Change wgServer and wgCanonicalServer for arbcom wikis [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/121709' (owner: 'Hoo man')
|
|
2014-04-08 18:45:06
|
<logmsgbot>
|
!log ori updated /a/common to {{Gerrit|I4b18e4ce8}}: Change wgServer and wgCanonicalServer for arbcom wikis
|
|
2014-04-08 18:45:11
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 18:45:28
|
<hoo>
|
heh :)
|
|
2014-04-08 18:45:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 18:45:50
|
<logmsgbot>
|
!log ori synchronized wmf-config/InitialiseSettings.php 'I4b18e4ce8: Change wgServer and wgCanonicalServer for arbcom wikis'
|
|
2014-04-08 18:45:55
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 18:53:40
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 18:56:09
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 18:57:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 172.800003
|
|
2014-04-08 18:58:59
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1012 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 18:59:00
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1001 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1895: active_shards: 5202: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 409
|
|
2014-04-08 18:59:00
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1009 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1895: active_shards: 5202: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 409
|
|
2014-04-08 18:59:00
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1004 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1895: active_shards: 5202: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 409
|
|
2014-04-08 18:59:00
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1010 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1895: active_shards: 5202: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 409
|
|
2014-04-08 18:59:09
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1013 is CRITICAL: CRITICAL - Could not connect to server 10.64.48.10
|
|
2014-04-08 18:59:09
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1003 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1895: active_shards: 5202: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 409
|
|
2014-04-08 18:59:09
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1006 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1895: active_shards: 5202: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 409
|
|
2014-04-08 18:59:10
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1016 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1895: active_shards: 5202: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 409
|
|
2014-04-08 18:59:29
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1015 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1895: active_shards: 5202: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 409
|
|
2014-04-08 19:00:03
|
<manybubbles>
|
blhe
|
|
2014-04-08 19:00:11
|
<manybubbles>
|
it recovered in a few seconds
|
|
2014-04-08 19:00:16
|
<manybubbles>
|
not sure why it did that
|
|
2014-04-08 19:07:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 341.200012
|
|
2014-04-08 19:12:00
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1001 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:12:00
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1009 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:12:00
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1004 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:12:00
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1010 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:12:10
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1003 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:12:11
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1013 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:12:11
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1016 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:12:11
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1006 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:13:16
|
<manybubbles>
|
thats right
|
|
2014-04-08 19:13:18
|
<manybubbles>
|
horrible check
|
|
2014-04-08 19:13:36
|
<manybubbles>
|
no errors in the logs associated with those warnings
|
|
2014-04-08 19:18:49
|
<icinga-wm>
|
RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000
|
|
2014-04-08 19:20:55
|
<huh>
|
https://en.wikipedia.org/wiki/Wikipedia:VPT#Heartbleed_bug.3F
|
|
2014-04-08 19:23:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 531.166687
|
|
2014-04-08 19:24:29
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1015 is CRITICAL: CRITICAL - Could not connect to server 10.64.48.12
|
|
2014-04-08 19:24:49
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1007 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:24:50
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1014 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:24:50
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1008 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:24:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1011 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:24:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1005 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:24:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1012 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:24:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1009 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:24:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1001 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:24:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1004 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1894: active_shards: 5197: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 414
|
|
2014-04-08 19:25:09
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 635.799988
|
|
2014-04-08 19:25:11
|
<Jamesofur>
|
kicks icinga-wm
|
|
2014-04-08 19:26:39
|
<icinga-wm>
|
PROBLEM - DPKG on elastic1015 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
|
|
2014-04-08 19:28:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 19:29:38
|
<matanya>
|
huh: it is being fixed by ops
|
|
2014-04-08 19:31:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 19:36:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 19:37:49
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1014 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:37:49
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1007 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:37:50
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1008 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:37:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1011 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:37:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1005 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:37:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1012 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:37:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1004 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:37:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1009 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:37:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1001 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:38:00
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1010 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:38:07
|
<huh>
|
again?
|
|
2014-04-08 19:38:09
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 19:38:10
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1013 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:38:10
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1003 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:38:10
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1016 is CRITICAL: CRITICAL - Could not connect to server 10.64.48.13
|
|
2014-04-08 19:38:10
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1006 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:38:29
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1015 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1894: active_shards: 5204: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 407
|
|
2014-04-08 19:38:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 224.199997
|
|
2014-04-08 19:39:39
|
<icinga-wm>
|
RECOVERY - DPKG on elastic1015 is OK: All packages OK
|
|
2014-04-08 19:40:19
|
<manybubbles>
|
oh shut up
|
|
2014-04-08 19:40:52
|
<manybubbles>
|
I'm doing rolling restarts
|
|
2014-04-08 19:41:47
|
<manybubbles>
|
got it: labswiki_content_1394813391
|
|
2014-04-08 19:41:53
|
<manybubbles>
|
that thing is configured without replicas
|
|
2014-04-08 19:46:40
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 341.066681
|
|
2014-04-08 19:48:00
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1004 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:01
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1009 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:01
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1001 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:01
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1010 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:10
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1003 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:10
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1013 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:10
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1006 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:10
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1016 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:30
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1015 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:43
|
<manybubbles>
|
and, more noise!
|
|
2014-04-08 19:48:49
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1007 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:49
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1014 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:49
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1008 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:48:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1005 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5308: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 303
|
|
2014-04-08 19:48:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1011 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5308: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 303
|
|
2014-04-08 19:48:59
|
<icinga-wm>
|
PROBLEM - ElasticSearch health check on elastic1012 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 15: number_of_data_nodes: 15: active_primary_shards: 1894: active_shards: 5308: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 303
|
|
2014-04-08 19:49:22
|
<manybubbles>
|
bit me labswiki!
|
|
2014-04-08 19:52:34
|
<bd808|LUNCH>
|
cheers manybubbles on
|
|
2014-04-08 19:52:53
|
<manybubbles>
|
it'll spam us again in a few minutes
|
|
2014-04-08 19:52:59
|
<manybubbles>
|
labswiki recovered a long time ago
|
|
2014-04-08 19:53:05
|
<manybubbles>
|
it was only out for ~30 seconds each time
|
|
2014-04-08 19:53:20
|
<manybubbles>
|
but ganglia wants all the shards on all the wikis to be recovered before it is happy
|
|
2014-04-08 19:53:59
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1005 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:53:59
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1011 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:53:59
|
<icinga-wm>
|
RECOVERY - ElasticSearch health check on elastic1012 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1896: active_shards: 5611: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0
|
|
2014-04-08 19:56:15
|
<manybubbles>
|
!log upgraded all elasticsearch servers except elastic1008. that is coming now.
|
|
2014-04-08 19:56:20
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 19:58:20
|
<manybubbles>
|
!log finished upgrading to Elasticsearch 1.1.0. The process went well with no issues other then some knocking out search in labs 3 times for 30 seconds a piece. And logging lots of nasty warnings to irc. I've started to the process to fix search in labs so it won't happen again.
|
|
2014-04-08 19:58:25
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 20:05:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 420.066681
|
|
2014-04-08 20:08:09
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 539.900024
|
|
2014-04-08 20:10:29
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 20:10:29
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 20:10:29
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 20:10:29
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 20:10:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 20:12:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 20:16:56
|
<se4598>
|
Does someone here know about dns issues with wmflabs-domains or related stuff that happened recently?
|
|
2014-04-08 20:19:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 20:20:41
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 176.399994
|
|
2014-04-08 20:22:09
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 20:26:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 368.466675
|
|
2014-04-08 20:28:02
|
<cajoel>
|
re:heartbleed, I think we'll be wanting a new corp certificate... do you guys have a favorite vendor for star certs these days?
|
|
2014-04-08 20:28:21
|
<cajoel>
|
it's almost due for a re-up anyway, so it's worth the effort
|
|
2014-04-08 20:29:53
|
<ebernhardson>
|
r
|
|
2014-04-08 20:48:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 642.700012
|
|
2014-04-08 20:51:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 20:51:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 20:52:09
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 537.099976
|
|
2014-04-08 20:59:46
|
<odder>
|
greg-g: don't believe you
|
|
2014-04-08 20:59:58
|
<odder>
|
http://lists.wikimedia.org/pipermail/wikitech-ambassadors/2014-April/000666.html
|
|
2014-04-08 21:00:04
|
<odder>
|
This is the work of the Beast
|
|
2014-04-08 21:00:11
|
<bd808>
|
greg-g: Do you still want to try group1 to 1.23wmf21 today or have we had enough excitement?
|
|
2014-04-08 21:00:53
|
<apergos>
|
reminds folks that all ops are out at a bar except for those who are about to go to sleep :-D
|
|
2014-04-08 21:01:06
|
<greg-g>
|
bd808: we're back to "if you run scap, run it twice" world, right?
|
|
2014-04-08 21:01:10
|
<greg-g>
|
apergos: :)
|
|
2014-04-08 21:01:23
|
<greg-g>
|
odder: which part? :)
|
|
2014-04-08 21:01:36
|
<bd808>
|
greg-g: Yes, but for group1 to 1.23wmf21 we only need to run sync-wikiversions
|
|
2014-04-08 21:01:49
|
<greg-g>
|
right
|
|
2014-04-08 21:02:09
|
<greg-g>
|
the world looks sane on phase0?
|
|
2014-04-08 21:02:11
|
<greg-g>
|
looks
|
|
2014-04-08 21:02:34
|
<odder>
|
greg-g: all of it - notice the number immediately preceding .html
|
|
2014-04-08 21:02:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 232.46666
|
|
2014-04-08 21:02:48
|
<greg-g>
|
odder: haha
|
|
2014-04-08 21:03:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 21:03:54
|
<greg-g>
|
this is neat: https://graphite.wikimedia.org/render/…
|
|
2014-04-08 21:04:36
|
<greg-g>
|
I think that's what ori told me yesterdayt to not worry about
|
|
2014-04-08 21:05:09
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 21:05:25
|
<greg-g>
|
bd808: if we do, we do now, so we have 2 hours before SWAT of settle bug report time. May I take your whole day?
|
|
2014-04-08 21:06:36
|
<bd808>
|
greg-g: I'm yours to command. :)
|
|
2014-04-08 21:06:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 269.866669
|
|
2014-04-08 21:06:42
|
<odder>
|
http://heartbleed.com/
|
|
2014-04-08 21:06:48
|
<odder>
|
Q&A
|
|
2014-04-08 21:06:55
|
<odder>
|
:-P
|
|
2014-04-08 21:07:09
|
<greg-g>
|
bd808: go forth, please
|
|
2014-04-08 21:09:36
|
<grrrit-wm>
|
('PS1') 'BryanDavis': Group1 wikis to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124744'
|
|
2014-04-08 21:11:12
|
<grrrit-wm>
|
('CR') 'BryanDavis': [C: '2'] Group1 wikis to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124744' (owner: 'BryanDavis')
|
|
2014-04-08 21:11:20
|
<grrrit-wm>
|
('Merged') 'jenkins-bot': Group1 wikis to 1.23wmf21 [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124744' (owner: 'BryanDavis')
|
|
2014-04-08 21:12:17
|
<logmsgbot>
|
!log bd808 rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to 1.23wmf21
|
|
2014-04-08 21:12:23
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 21:12:47
|
<hoo>
|
greg-g: Have you guys already killed all user sessions?
|
|
2014-04-08 21:12:52
|
<hoo>
|
Can't see a server admin log entry
|
|
2014-04-08 21:15:44
|
<odder>
|
greg-g: I did a https://commons.wikimedia.org/wiki/Commons:Village_pump#Users_are_being_forced_to_log_out
|
|
2014-04-08 21:18:21
|
<Jamesofur>
|
Thanks odder, I left a note about it on en VPT since I saw a question about the bug in general
|
|
2014-04-08 21:18:48
|
<odder>
|
Maybe I'll cross-post that to Meta too
|
|
2014-04-08 21:19:59
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 21:20:14
|
<logmsgbot>
|
!log bd808 Purged l10n cache for 1.23wmf18
|
|
2014-04-08 21:20:19
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 21:21:46
|
<logmsgbot>
|
!log bd808 Purged l10n cache for 1.23wmf19
|
|
2014-04-08 21:21:50
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 21:21:54
|
<greg-g>
|
hoo: in process
|
|
2014-04-08 21:22:55
|
<hoo>
|
:)
|
|
2014-04-08 21:23:09
|
<greg-g>
|
hoo: it takes longer than you'd imagine, maybe :)
|
|
2014-04-08 21:23:37
|
<bd808|deploy>
|
greg-g: group1 to 1.23wmf21 is {{done}}
|
|
2014-04-08 21:23:40
|
<se4598>
|
greg-g: just change the cookie name? (like last time)
|
|
2014-04-08 21:24:09
|
<greg-g>
|
se4598: I'm defering to chris on it (not sure what his exact process is, honestly)
|
|
2014-04-08 21:24:14
|
<greg-g>
|
bd808|deploy: ty
|
|
2014-04-08 21:24:53
|
<se4598>
|
mh, the tokens will be still valid I think, wasn't a good idea
|
|
2014-04-08 21:25:14
|
<bd808>
|
se4598: Yeah I think that's why it takes a while
|
|
2014-04-08 21:26:45
|
<hoo>
|
greg-g: Well given how many users we have and that we probably don't want to hammer the DBs to much, I can imagine this to take some time
|
|
2014-04-08 21:26:52
|
<greg-g>
|
nods
|
|
2014-04-08 21:28:16
|
<hoo>
|
csteipp: Why not run one process per shard?
|
|
2014-04-08 21:29:24
|
<odder>
|
Jamesofur: if you're keeping track of things, I alerted Commons and Meta; perhaps someone would need to alert the other big Wikipedias
|
|
2014-04-08 21:29:35
|
<odder>
|
Dunno if the message to tech-ambassadors will be enough; may be.
|
|
2014-04-08 21:30:35
|
<grrrit-wm>
|
('PS2') 'MaxSem': Put a safeguard on GeoData's usage of CirrusSearch [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/121874'
|
|
2014-04-08 21:30:37
|
<grrrit-wm>
|
('PS1') 'MaxSem': Enable $wgGeoDataDebug on labs [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124747'
|
|
2014-04-08 21:30:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 21:30:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 535.0
|
|
2014-04-08 21:30:54
|
<grrrit-wm>
|
('CR') 'jenkins-bot': [V: '-1'] Enable $wgGeoDataDebug on labs [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124747' (owner: 'MaxSem')
|
|
2014-04-08 21:31:32
|
<csteipp>
|
se4598: Assuming attacker has the login token, they could use the new name and again spoof the user
|
|
2014-04-08 21:31:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 21:31:46
|
<grrrit-wm>
|
('PS2') 'MaxSem': Enable $wgGeoDataDebug on labs [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124747'
|
|
2014-04-08 21:32:09
|
<Jamesofur>
|
odder: yeah, I'll see if we can poke people, we're going to send out SM messages as well in a couple minutes
|
|
2014-04-08 21:32:19
|
<Jamesofur>
|
with a recommendation to password reset
|
|
2014-04-08 21:33:09
|
<odder>
|
SM?
|
|
2014-04-08 21:33:22
|
<Jamesofur>
|
sorry, Social Media (Twitter/Facebook/G+ etc)
|
|
2014-04-08 21:33:42
|
<odder>
|
TMA, Too Many Abbreviations
|
|
2014-04-08 21:33:45
|
<odder>
|
:)
|
|
2014-04-08 21:33:59
|
<Jamesofur>
|
yup lol
|
|
2014-04-08 21:34:09
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 539.133362
|
|
2014-04-08 21:34:10
|
<Jamesofur>
|
I abuse them, I even make up my own and forget that they are just in my head
|
|
2014-04-08 21:34:23
|
<HaeB>
|
https://twitter.com/Wikimedia/status/453646877397757953
|
|
2014-04-08 21:34:49
|
<JohnLewis>
|
Jamesofur: EUS IAA. TA IANAL.
|
|
2014-04-08 21:34:58
|
<JohnLewis>
|
*EYS :p
|
|
2014-04-08 21:35:42
|
<odder>
|
thanks HaeB, retweeted
|
|
2014-04-08 21:40:46
|
<aude>
|
woah, new code on wikidata?
|
|
2014-04-08 21:40:46
|
<matanya>
|
Jamesofur: using mass-message might be a good idea
|
|
2014-04-08 21:41:15
|
<greg-g>
|
aude: yep, all ok?
|
|
2014-04-08 21:41:26
|
<Jamesofur>
|
HaeB: ^ what do you think? (about MM)
|
|
2014-04-08 21:41:48
|
<greg-g>
|
wdyt?
|
|
2014-04-08 21:42:08
|
<JohnLewis>
|
greg-g: itjdi
|
|
2014-04-08 21:42:12
|
<aude>
|
so we're confident?
|
|
2014-04-08 21:42:39
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3012 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 187.866669
|
|
2014-04-08 21:42:53
|
<greg-g>
|
aude: in that it won't break at 2:00 utc? yeah
|
|
2014-04-08 21:43:06
|
<greg-g>
|
aude: the only thing we're still not confident about is scap on thursday
|
|
2014-04-08 21:44:19
|
<aude>
|
alright
|
|
2014-04-08 21:44:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 21:44:40
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 320.200012
|
|
2014-04-08 21:44:55
|
<HaeB>
|
Jamesofur, matanya: i think for the session ending, massmessage would be overkill. regarding the password reset, it's a judgment call (how high one estimates the risk for users who don't change it)
|
|
2014-04-08 21:45:24
|
<matanya>
|
HaeB: it depends on user rights as well
|
|
2014-04-08 21:45:27
|
<bd808>
|
aude: The bug that caused all the 1.23wmf21 l10n issues is https://bugzilla.wikimedia.org/show_bug.cgi?id=63659
|
|
2014-04-08 21:46:31
|
<HaeB>
|
are there any other major sites who notified all users?
|
|
2014-04-08 21:46:54
|
<Jamesofur>
|
not that I've seen yet, but I have a feeling some are still going through the fixing process
|
|
2014-04-08 21:46:55
|
<aude>
|
interesting
|
|
2014-04-08 21:46:59
|
<HaeB>
|
(to recommend a password chanage)
|
|
2014-04-08 21:47:10
|
<hoo>
|
eg. just got stuff from CloudBees
|
|
2014-04-08 21:47:15
|
<hoo>
|
github also logged me out
|
|
2014-04-08 21:47:37
|
<HaeB>
|
would also be interesting to know how quick the wikis were fixed after the news broke yesterday
|
|
2014-04-08 21:47:40
|
<Jamesofur>
|
latimes has an article about resetting your password, but that's different
|
|
2014-04-08 21:48:09
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 21:48:13
|
<HaeB>
|
last night (PT) i filed a RT ticket for the blog, which was vulnerable at the time, but at that point the wikis tested ok already
|
|
2014-04-08 21:48:36
|
<hoo>
|
The wikis auto update OpenSSL via puppet
|
|
2014-04-08 21:49:00
|
<Jamesofur>
|
hoo: well ya ;) the question is when we updated puppet ;)
|
|
2014-04-08 21:49:24
|
<hoo>
|
Jamesofur: The servers do that themselves
|
|
2014-04-08 21:49:39
|
<HaeB>
|
per https://wikitech.wikimedia.org/wiki/Server_admin_log , the blog (holmium) was pretty late in the game
|
|
2014-04-08 21:49:50
|
<bd808>
|
The timeline is all in SAL from last night
|
|
2014-04-08 21:49:51
|
<hoo>
|
Yesterday I posted about that to the internal ops list, but forgot to poke a root to do a apt-cache clean and force puppet run
|
|
2014-04-08 21:50:08
|
<HaeB>
|
"04:03 Tim: upgrading libssl on ssl1001,ssl1002,ssl1003,ssl1004,ssl1005,ssl1006,ssl1007,ssl1008,ssl1009,ssl3001.esams.wikimedia.org,ssl3002.esams.wikimedia.org,ssl3003.esams.wikimedia.org" - is that the entry for the wikis?
|
|
2014-04-08 21:50:37
|
<bd808>
|
Mostly yes
|
|
2014-04-08 21:53:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3012 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 21:53:39
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 21:53:59
|
<icinga-wm>
|
RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000
|
|
2014-04-08 21:54:55
|
<grrrit-wm>
|
('PS1') 'Jean-Frédéric': Add Musées de la Haute-Saône to wgCopyUploadsDomains [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124754'
|
|
2014-04-08 22:01:11
|
<mwalker>
|
greg-g, poking you because I'm not sure who's on point for the i18n / scap stuff -- but I recall getting pinged a couple of days ago (on a centralnotice keyword) saying that the i18n update was failing due to exceptions on CN (and others). I'm wondering if CN's fail was due to being on a deployment branch that did not have the JSON updates (until just now).
|
|
2014-04-08 22:01:46
|
<greg-g>
|
shouldn't be
|
|
2014-04-08 22:01:57
|
<greg-g>
|
there's backward compat in l10nupdate
|
|
2014-04-08 22:02:17
|
<greg-g>
|
mwalker: see https://bugzilla.wikimedia.org/show_bug.cgi?id=63659 for all the gorey details
|
|
2014-04-08 22:02:33
|
<mwalker>
|
puts on tyvek suit
|
|
2014-04-08 22:02:38
|
<greg-g>
|
:)
|
|
2014-04-08 22:30:59
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 22:33:06
|
<csteipp>
|
greg-g: Could I push a small centralauth update soon?
|
|
2014-04-08 22:33:44
|
<greg-g>
|
yeah, now is fine, 30 minutes until swat
|
|
2014-04-08 22:34:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:36:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:37:04
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 34.533333
|
|
2014-04-08 22:37:34
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 260.733337
|
|
2014-04-08 22:38:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:40:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:42:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:44:14
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 625.166687
|
|
2014-04-08 22:44:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:45:36
|
<se4598>
|
marktraceur: I see in deploy-calendar that you have changeset which especially activates MediaViewer on en-beta. You(r pc) may get hit by https://bugzilla.wikimedia.org/show_bug.cgi?id=63709
|
|
2014-04-08 22:46:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:47:22
|
<marktraceur>
|
se4598: Is there a fix?
|
|
2014-04-08 22:47:50
|
<marktraceur>
|
I'm guessing it's an SSL problem
|
|
2014-04-08 22:48:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:48:43
|
<marktraceur>
|
se4598: Replied on bug
|
|
2014-04-08 22:49:09
|
<grrrit-wm>
|
('PS1') 'BryanDavis': Create symlink for compile-wikiversions in /usr/local/bin [operations/puppet] - 'https://gerrit.wikimedia.org/r/124763'
|
|
2014-04-08 22:49:23
|
<se4598>
|
marktraceur: We in #wikimedia-labs haven't one. And thats not about https but dns resolve, so I don't understand what do you mean by https?
|
|
2014-04-08 22:49:35
|
<marktraceur>
|
Oh, hm
|
|
2014-04-08 22:49:37
|
<marktraceur>
|
Never mind, sorry
|
|
2014-04-08 22:50:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:52:04
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 22:52:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:52:34
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 22:52:56
|
<se4598>
|
marktraceur: currently the fix is.....: it may work if you try multiple times or wait some time (minutes, hours) ;P
|
|
2014-04-08 22:54:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:56:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:56:54
|
<icinga-wm>
|
RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000
|
|
2014-04-08 22:58:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 10:30:08 PM UTC
|
|
2014-04-08 22:58:41
|
<hoo>
|
greg-g: csteipp: got both core changes ready
|
|
2014-04-08 22:58:53
|
<hoo>
|
I mean changes to the deploy branch
|
|
2014-04-08 22:59:52
|
<csteipp>
|
hoo: Cool.. one sec and I'll merge and deploy it
|
|
2014-04-08 23:00:12
|
<hoo>
|
I can also jump in, am on tin still anyway
|
|
2014-04-08 23:00:14
|
<icinga-wm>
|
RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Tue Apr 8 23:00:04 UTC 2014
|
|
2014-04-08 23:02:24
|
<icinga-wm>
|
PROBLEM - Puppet freshness on mw1109 is CRITICAL: Last successful Puppet run was Tue 08 Apr 2014 11:00:04 PM UTC
|
|
2014-04-08 23:05:24
|
<greg-g>
|
stupid puppet
|
|
2014-04-08 23:06:33
|
<Jasper_Deng>
|
always wondered what Puppet does anyways
|
|
2014-04-08 23:07:09
|
<Jamesofur>
|
pulls the strings ;)
|
|
2014-04-08 23:07:20
|
<Jamesofur>
|
(or, probably better 'is the strings' )
|
|
2014-04-08 23:07:26
|
<hoo>
|
Jasper_Deng: Playing with the servers :D
|
|
2014-04-08 23:08:20
|
<JohnLewis>
|
Technically, the sysadmins are a puppet in the WMFs plans, right? :p
|
|
2014-04-08 23:08:37
|
<logmsgbot>
|
!log csteipp synchronized php-1.23wmf21/extensions/CentralAuth/maintenance 'Push maintenance script for token reset'
|
|
2014-04-08 23:08:39
|
<Jamesofur>
|
or we're all just puppets in their plans, duh
|
|
2014-04-08 23:08:41
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 23:09:04
|
<JohnLewis>
|
Jamesofur: You're the past of the puppets :p
|
|
2014-04-08 23:09:09
|
<JohnLewis>
|
*master of the
|
|
2014-04-08 23:09:57
|
<csteipp>
|
greg-g: CentralAuth updates are out, so swat can go ahead if they were waiting on me
|
|
2014-04-08 23:10:01
|
<Jamesofur>
|
;) the user with said name may dislike me claiming the title
|
|
2014-04-08 23:10:40
|
<greg-g>
|
mwalker: ori ebernhardson ^
|
|
2014-04-08 23:10:46
|
<greg-g>
|
also, what the heck, oit_display ?
|
|
2014-04-08 23:10:54
|
<greg-g>
|
:)
|
|
2014-04-08 23:11:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 23:11:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 23:11:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 23:11:10
|
<icinga-wm>
|
PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC
|
|
2014-04-08 23:11:51
|
<mwalker>
|
oh
|
|
2014-04-08 23:11:54
|
<mwalker>
|
yes; it's 4!
|
|
2014-04-08 23:13:25
|
<Danny_B>
|
SUL doesn't work?
|
|
2014-04-08 23:14:02
|
<mwalker>
|
csteipp, ^
|
|
2014-04-08 23:14:03
|
<hoo>
|
Danny_B: We are logging out all users
|
|
2014-04-08 23:14:10
|
<hoo>
|
see http://lists.wikimedia.org/pipermail/wikitech-ambassadors/2014-April/000666.html
|
|
2014-04-08 23:14:32
|
<MaxSem>
|
csteipp, warn ppl with a site notice?
|
|
2014-04-08 23:14:35
|
<se4598>
|
hoo: you know that this isn't merged? https://gerrit.wikimedia.org/r/124756
|
|
2014-04-08 23:15:00
|
<hoo>
|
se4598: not this important at the very moments
|
|
2014-04-08 23:15:03
|
<hoo>
|
* moment
|
|
2014-04-08 23:15:23
|
<csteipp>
|
Danny_B: SUL should work... You should just be logged out. If you can't login, let me know
|
|
2014-04-08 23:15:53
|
<Jamesofur>
|
csteipp: will we get logged out each time you hit a wiki we've visited recently? or just the once per user in theory
|
|
2014-04-08 23:16:15
|
<csteipp>
|
If you're a global user, just once (right now as I logout all the centralauth users)
|
|
2014-04-08 23:16:32
|
<csteipp>
|
If you have multiple ununified local accounts, each will get logged out
|
|
2014-04-08 23:16:51
|
<Danny_B>
|
csteipp: i have to log in on every single project although i have central username
|
|
2014-04-08 23:16:54
|
<Amgine>
|
<grumbles about that><waves fist impotently at it.wp>
|
|
2014-04-08 23:17:30
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 135.300003
|
|
2014-04-08 23:17:55
|
<mwalker>
|
marktraceur, MaxSem I'm going to +2 and confirm https://gerrit.wikimedia.org/r/#/c/124036/2 , https://gerrit.wikimedia.org/r/#/c/121874/2 , https://gerrit.wikimedia.org/r/#/c/124747/
|
|
2014-04-08 23:18:30
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 173.666672
|
|
2014-04-08 23:18:32
|
<mwalker>
|
it would be wonderful if you all could +1 that so that I know you've looked and said this is good to me
|
|
2014-04-08 23:18:35
|
<marktraceur>
|
'kay
|
|
2014-04-08 23:18:53
|
<Danny_B>
|
csteipp: +1 to notice ppl with central notice
|
|
2014-04-08 23:18:57
|
<grrrit-wm>
|
('CR') 'MarkTraceur': [C: ''] Add setting to show a survey for MediaViewer users on some sites [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124036' (owner: 'Gerg? Tisza')
|
|
2014-04-08 23:19:00
|
<MaxSem>
|
+1 ourselves?
|
|
2014-04-08 23:19:16
|
<MaxSem>
|
doesn't sound very assuring:)
|
|
2014-04-08 23:19:21
|
<mwalker>
|
nah; you're probably OK MaxSem :p
|
|
2014-04-08 23:19:27
|
<mwalker>
|
but I don't know who Gergo is
|
|
2014-04-08 23:19:44
|
<mwalker>
|
but mark was sponsoring the patch
|
|
2014-04-08 23:19:53
|
<MaxSem>
|
he's tgr :P
|
|
2014-04-08 23:20:00
|
<grrrit-wm>
|
('CR') 'Mwalker': [C: '2'] Put a safeguard on GeoData's usage of CirrusSearch [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/121874' (owner: 'MaxSem')
|
|
2014-04-08 23:20:08
|
<grrrit-wm>
|
('CR') 'Mwalker': [C: '2'] Enable $wgGeoDataDebug on labs [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124747' (owner: 'MaxSem')
|
|
2014-04-08 23:20:21
|
<grrrit-wm>
|
('CR') 'Mwalker': [C: '2'] Add setting to show a survey for MediaViewer users on some sites [operations/mediawiki-config] - 'https://gerrit.wikimedia.org/r/124036' (owner: 'Gerg? Tisza')
|
|
2014-04-08 23:20:27
|
<ori>
|
greg-g: missed your ping; still need me?
|
|
2014-04-08 23:21:00
|
<greg-g>
|
dont think so
|
|
2014-04-08 23:23:33
|
<mwalker>
|
interesting; sync-common doesn't log to IRC?
|
|
2014-04-08 23:23:34
|
<csteipp>
|
Danny_B: That doesn't sound right.. At the risk of sounding cliche, can you log out and log back in, and see if that helps?
|
|
2014-04-08 23:23:55
|
<mwalker>
|
marktraceur, MaxSem can you tell if your configuration stuff got pushed?
|
|
2014-04-08 23:24:15
|
<MaxSem>
|
mwalker, mine's noop on prod
|
|
2014-04-08 23:24:25
|
<marktraceur>
|
Ditto, but will check on beta
|
|
2014-04-08 23:24:26
|
<MaxSem>
|
checking if prod still works...
|
|
2014-04-08 23:24:35
|
<mwalker>
|
also; marktraceur I presume you want https://gerrit.wikimedia.org/r/#/c/124510/ to go to wmf20 and wmf21?
|
|
2014-04-08 23:24:38
|
<HaeB>
|
Danny_B, hoo : we're still thinking about massmessage instead (more for the password changing advice)
|
|
2014-04-08 23:24:43
|
<marktraceur>
|
mwalker: Sorry, only 21
|
|
2014-04-08 23:25:24
|
<marktraceur>
|
mwalker: Confirmed, beta has the configuration we wanted
|
|
2014-04-08 23:26:36
|
<MaxSem>
|
mwalker, lgtm
|
|
2014-04-08 23:27:40
|
<icinga-wm>
|
PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: No output from Graphite for target(s): reqstats.5xx
|
|
2014-04-08 23:28:34
|
<Danny_B>
|
csteipp: log out from any currently logged project, log back to it and then try if sul works on other?
|
|
2014-04-08 23:29:14
|
<csteipp>
|
Danny_B: Yeah
|
|
2014-04-08 23:29:22
|
<Danny_B>
|
csteipp: ok, sec
|
|
2014-04-08 23:29:38
|
<csteipp>
|
Hmm... Danny_B What's you're wiki username?
|
|
2014-04-08 23:30:51
|
<icinga-wm>
|
RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Tue Apr 8 23:30:43 UTC 2014
|
|
2014-04-08 23:30:55
|
<Danny_B>
|
csteipp: Danny B.
|
|
2014-04-08 23:31:17
|
<Danny_B>
|
csteipp: seems to work now, will let you know if i'll spot another disconnection
|
|
2014-04-08 23:31:27
|
<csteipp>
|
Danny_B: Cool, thanks
|
|
2014-04-08 23:32:03
|
<Danny_B>
|
yw
|
|
2014-04-08 23:32:15
|
<Danny_B>
|
thanks for care
|
|
2014-04-08 23:33:30
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 23:34:31
|
<logmsgbot>
|
!log mwalker synchronized php-1.23wmf21/extensions/MultimediaViewer/ 'Updating MultimediaViewer for {{gerrit|124510}}'
|
|
2014-04-08 23:34:35
|
<morebots>
|
Logged the message, Master
|
|
2014-04-08 23:35:16
|
<mwalker>
|
marktraceur, ^ if you would test what you need to test for that
|
|
2014-04-08 23:35:26
|
<mwalker>
|
I'm not seeing any fatals or exceptions which is good :)
|
|
2014-04-08 23:35:31
|
<icinga-wm>
|
RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0
|
|
2014-04-08 23:35:32
|
<marktraceur>
|
mwalker: Works
|
|
2014-04-08 23:35:32
|
<marktraceur>
|
Ta
|
|
2014-04-08 23:35:39
|
<mwalker>
|
cool; greg-g SWAT done
|
|
2014-04-08 23:58:30
|
<icinga-wm>
|
PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 179.666672
|
|
2014-04-08 23:59:04
|
<jackmcbarn>
|
"Firefox can't find the server at en.wikipedia.beta.wmflabs.org."
|
|
2014-04-08 23:59:08
|
<jackmcbarn>
|
why?
|
|
2014-04-08 23:59:14
|
<grrrit-wm>
|
('CR') 'Aaron Schulz': [C: ''] Create symlink for compile-wikiversions in /usr/local/bin [operations/puppet] - 'https://gerrit.wikimedia.org/r/124763' (owner: 'BryanDavis')
|
|
2014-04-08 23:59:31
|
<marktraceur>
|
jackmcbarn: https://bugzilla.wikimedia.org/show_bug.cgi?id=63709 probably
|