[00:01:46] <nagios-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:10:37] <nagios-wm_>	 PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours
[00:15:43] <nagios-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.262 seconds
[00:25:46] <nagios-wm_>	 PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 204 seconds
[00:50:13] <nagios-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:02:49] <nagios-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.471 seconds
[01:04:19] <nagios-wm_>	 PROBLEM - MySQL disk space on storage3 is CRITICAL: DISK CRITICAL - /archive is not accessible: Success
[01:34:37] <nagios-wm_>	 PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours
[01:34:37] <nagios-wm_>	 PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours
[01:38:40] <nagios-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:54:16] <nagios-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.021 seconds
[02:25:49] <logmsgbot_>	 !log LocalisationUpdate completed (1.21wmf1) at Tue Oct  9 02:25:49 UTC 2012
[02:26:09] <morebots>	 Logged the message, Master
[02:26:58] <nagios-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:37:46] <nagios-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds
[02:50:29] <logmsgbot_>	 !log LocalisationUpdate completed (1.20wmf12) at Tue Oct  9 02:50:29 UTC 2012
[02:50:41] <morebots>	 Logged the message, Master
[03:49:10] <nagios-wm_>	 RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds
[04:13:37] <nagios-wm_>	 PROBLEM - Router interfaces on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, interfaces up: 73, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-1/1/0: down - Core:  cr2-eqiad:xe-5/2/1 (FPL/Level3, CV71028) {#2163} [10Gbps wave]BR
[04:23:04] <nagios-wm_>	 RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 75, down: 0, dormant: 0, excluded: 0, unused: 0
[04:27:43] <nagios-wm_>	 PROBLEM - Router interfaces on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, interfaces up: 73, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-1/1/0: down - Core:  cr2-eqiad:xe-5/2/1 (FPL/Level3, CV71028) {#2163} [10Gbps wave]BR
[04:37:10] <nagios-wm_>	 RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 75, down: 0, dormant: 0, excluded: 0, unused: 0
[05:46:51] <gerrit-wm>	 New patchset: Tim Starling; "run mergeMessageFileList.php as 'mwdeploy' user" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27060
[05:47:49] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27060
[05:48:44] <gerrit-wm>	 New patchset: Tim Starling; "(bug 39701) beta: automatic MediaWiki update" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/22116
[05:49:37] <gerrit-wm>	 Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/22116
[05:58:37] <nagios-wm_>	 PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours
[05:58:37] <nagios-wm_>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[05:58:37] <nagios-wm_>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[06:32:40] <nagios-wm_>	 PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours
[06:50:38] <nagios-wm_>	 PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours
[07:31:44] <nagios-wm_>	 PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours
[07:59:38] <nagios-wm_>	 PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours
[08:48:41] <nagios-wm_>	 PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours
[09:25:31] <logmsgbot_>	 !log nikerabbit synchronized php-1.21wmf1/extensions/Translate/api/ApiGroupReview.php
[09:25:43] <morebots>	 Logged the message, Master
[09:31:32] <mukty>	 cara memperbesar penis
[09:32:55] <logmsgbot_>	 !log nikerabbit synchronized wmf-config/InitialiseSettings.php  'Bug 40366 - Narayam'
[09:33:06] <morebots>	 Logged the message, Master
[09:36:06] <paravoid>	 lol?
[10:11:38] <nagios-wm_>	 PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours
[10:15:14] <nagios-wm_>	 PROBLEM - check_job_queue on spence is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , itwiki (34029)
[10:15:32] <nagios-wm_>	 PROBLEM - check_job_queue on neon is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , itwiki (33319)
[10:46:26] <nagios-wm_>	 RECOVERY - check_job_queue on spence is OK: JOBQUEUE OK - all job queues below 10,000
[10:46:44] <nagios-wm_>	 RECOVERY - check_job_queue on neon is OK: JOBQUEUE OK - all job queues below 10,000
[10:59:27] <paravoid>	 !log rolling restart of swift proxies (ms-fe2/3/4) because of the memory leaks
[10:59:40] <morebots>	 Logged the message, Master
[11:16:52] <mark>	 time to setup snapmirror
[11:25:41] <paravoid>	 yay
[11:25:49] <paravoid>	 want me to do something?
[11:27:09] <mark>	 get me coffee?
[11:27:40] <mark>	 !log Created 36 drive media1 raid-dp aggregate on nas1001-a, hosting new 40T volume 'originals'
[11:27:51] <morebots>	 Logged the message, Master
[11:28:22] <paravoid>	 I'm a bit far but if you insist
[11:29:00] <paravoid>	 you'll have to wait though, both the city center and all roads to the airport are closed
[11:29:18] <mark>	 show some ambition
[11:29:21] <paravoid>	 chancellor merkel decided to do us a surprise visit and it's... interesting around here :)
[11:29:26] <mark>	 yeah I saw
[11:29:35] <mark>	 !log Increased maxfiles of nas1001-a:originals to 63753378
[11:29:46] <morebots>	 Logged the message, Master
[11:35:08] <mark>	 !log Setup sync SnapMirror relationship from nas1-a:originals to nas1001-a:originals and started transfer
[11:35:19] <morebots>	 Logged the message, Master
[11:35:38] <nagios-wm_>	 PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours
[11:35:38] <nagios-wm_>	 PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours
[11:50:33] <mark>	 50 GB done
[11:52:44] <nagios-wm_>	 PROBLEM - Puppet freshness on sq69 is CRITICAL: Puppet has not run in the last 10 hours
[11:54:58] <paravoid>	 media say that Merkel is wearing the same jacket that she wore on the Greece-Germany Euro 2012 game
[11:55:04] <paravoid>	 *sigh*
[11:55:52] <mark>	 depicting her like hitler doesn't really help
[11:57:28] <paravoid>	 no it doesn't
[11:59:56] <nagios-wm_>	 PROBLEM - check_job_queue on spence is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , enwiki (13374)
[12:15:32] <nagios-wm_>	 RECOVERY - check_job_queue on spence is OK: JOBQUEUE OK - all job queues below 10,000
[12:16:05] <hashar>	 hello
[12:18:21] <paravoid>	 hi
[12:51:23] <nagios-wm_>	 PROBLEM - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL - No data received from host
[12:52:53] <nagios-wm_>	 RECOVERY - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is OK: HTTP OK HTTP/1.1 200 OK - 59624 bytes in 0.167 seconds
[12:53:08] <paravoid>	 wtf
[12:54:08] <paravoid>	 can't see anything wrong
[12:54:47] <hashar>	 hi paravoid :)
[12:54:54] <paravoid>	 hi
[12:55:06] <hashar>	 if you see nothing, then it is probably fine :-]
[13:31:47] <logmsgbot_>	 !log reedy synchronized wmf-config/CommonSettings.php  'Pre-emptively set $wgContentHandlerUseDB = false;'
[13:31:58] <morebots>	 Logged the message, Master
[14:30:43] <cmjohnson1>	 jeff_green: storage3 has another bad disk (120
[14:31:26] <nagios-wm_>	 PROBLEM - check_job_queue on spence is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , enwiki (32650)
[14:36:14] <nagios-wm_>	 PROBLEM - check_job_queue on neon is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , enwiki (32470)
[14:58:53] <nagios-wm_>	 PROBLEM - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL - No data received from host
[15:00:23] <nagios-wm_>	 RECOVERY - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is OK: HTTP OK HTTP/1.1 200 OK - 59830 bytes in 0.192 seconds
[15:03:40] <gerrit-wm>	 New patchset: Demon; "cleanup/refactor gerrit logging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8120
[15:04:36] <gerrit-wm>	 New review: Demon; "Rebased this on top of Antoine's changes, since they're likely to go in first." [operations/puppet] (production) C: 0;  - https://gerrit.wikimedia.org/r/8120
[15:04:36] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/8120
[15:05:10] <gerrit-wm>	 New review: Demon; "This is going to need rebasing, but let's *please* get this in soon. I'm tired of waiting for it :)" [operations/puppet] (production) C: 0;  - https://gerrit.wikimedia.org/r/8344
[15:20:05] <paravoid>	 https://gdash.wikimedia.org/dashboards/parser/ looks like an anomaly
[15:25:43] <mark>	 indeed it does
[15:30:06] <paravoid>	 AaronSchulz: around?
[15:30:15] <paravoid>	 it's a bit early, but you never know
[15:32:03] <Reedy>	 I saw him close a bug a couple of hours ago..
[15:43:34] <paravoid>	 got him, he has two different nicknames and the home one isn't here but is on -tech :)
[15:51:41] <Jeff_Green>	 cmjohnson1: re storage3, why am I not surprised :-P
[15:52:33] <cmjohnson1>	 jeff_green: so I was looking at the hard drives and disk 4 and 13 are unconfigured but good
[15:52:43] <cmjohnson1>	 i didn't want to mess with them unit l talked to you
[15:56:31] <gerrit-wm>	 New review: Pyoungmeister; "so, I'm generally very in favor of this transition, but I'm a bit concerned that some of these vars ..." [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/26957
[15:57:28] <gerrit-wm>	 New review: Mark Bergsma; "No way we're gonna merge that as one huge commit" [operations/puppet] (production); V: 0 C: -2;  - https://gerrit.wikimedia.org/r/26957
[15:59:38] <nagios-wm_>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[15:59:38] <nagios-wm_>	 PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours
[15:59:38] <nagios-wm_>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[16:01:28] <Jeff_Green>	 cmjohnson1: you can beat on it. it's already crashed anyawy
[16:02:04] <cmjohnson1>	 ok
[16:02:06] <cmjohnson1>	 cool
[16:02:27] <Jeff_Green>	 the fail rate on this machine is pretty amazing
[16:03:22] <Jeff_Green>	 sorta makes me wonder if there's a bad power supply or microwave oven on the same circuit
[16:06:03] <cmjohnson1>	 jeff_green: now that we are able to order through dell again, robh should be able to get the new raid controller card for the new storage) (db78)
[16:06:39] <Jeff_Green>	 ok
[16:06:57] <gerrit-wm>	 Change abandoned: Demon; "Abandoning per mark." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/26957
[16:18:01] <logmsgbot_>	 !log aaron synchronized wmf-config/filebackend.php  'Set abbrvThreshold to 200 for thumbnails.'
[16:18:13] <morebots>	 Logged the message, Master
[16:22:33] <logmsgbot_>	 !log aaron synchronized wmf-config/filebackend.php  'Set abbrvThreshold to 180 for thumbnails.'
[16:22:44] <morebots>	 Logged the message, Master
[16:27:52] <logmsgbot_>	 !log aaron synchronized wmf-config/filebackend.php  'Set abbrvThreshold to 160 for thumbnails.'
[16:28:03] <morebots>	 Logged the message, Master
[16:33:41] <nagios-wm_>	 PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours
[16:36:14] <nagios-wm_>	 RECOVERY - check_job_queue on spence is OK: JOBQUEUE OK - all job queues below 10,000
[16:41:02] <nagios-wm_>	 RECOVERY - check_job_queue on neon is OK: JOBQUEUE OK - all job queues below 10,000
[16:51:41] <nagios-wm_>	 PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours
[16:56:00] <gerrit-wm>	 New patchset: Andrew Bogott; "Use --passfile for the original mw admin pass." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27304
[16:57:05] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/27304
[16:57:32] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27304
[17:28:28] <notpeter>	 !log reimaging mw61 with precise for bits test
[17:28:39] <morebots>	 Logged the message, notpeter
[17:32:20] <nagios-wm_>	 PROBLEM - Host mw61 is DOWN: PING CRITICAL - Packet loss = 100%
[17:32:38] <nagios-wm_>	 PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours
[17:37:24] <gerrit-wm>	 New patchset: Pyoungmeister; "setting mw61 to use role::applicationserver::appserver::bits" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27310
[17:38:02] <nagios-wm_>	 RECOVERY - Host mw61 is UP: PING OK - Packet loss = 0%, RTA = 2.13 ms
[17:38:27] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/27310
[17:41:29] <nagios-wm_>	 PROBLEM - SSH on mw61 is CRITICAL: Connection refused
[17:41:29] <nagios-wm_>	 PROBLEM - Apache HTTP on mw61 is CRITICAL: Connection refused
[17:46:17] <nagios-wm_>	 RECOVERY - SSH on mw61 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[17:46:48] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27310
[17:47:34] <notpeter>	 merging various things on sockpuppet, including stuff for beta
[17:49:44] <nagios-wm_>	 PROBLEM - check_job_queue on spence is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , zhwiki (70471)
[17:54:05] <nagios-wm_>	 RECOVERY - Apache HTTP on mw61 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.007 seconds
[17:54:32] <nagios-wm_>	 PROBLEM - check_job_queue on neon is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , zhwiki (68966)
[17:57:19] <gerrit-wm>	 New review: Matthias Mullie; "Any chance anyone could merge this in by Thursday? I'd like to deploy this Thursday morning ;)" [operations/mediawiki-config] (master); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/25855
[17:59:56] <nagios-wm_>	 PROBLEM - NTP on mw61 is CRITICAL: NTP CRITICAL: Offset unknown
[18:00:41] <nagios-wm_>	 PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours
[18:02:29] <nagios-wm_>	 PROBLEM - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL - No data received from host
[18:03:59] <nagios-wm_>	 RECOVERY - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is OK: HTTP OK HTTP/1.1 200 OK - 59956 bytes in 0.180 seconds
[18:04:31] <binasher>	 rendering.. what happened?
[18:05:49] <hashar>	 andrewbogott: hey :)
[18:06:09] <andrewbogott>	 hashar:  howdy
[18:06:20] <apergos>	 dunno
[18:06:23] <nagios-wm_>	 PROBLEM - Apache HTTP on mw61 is CRITICAL: Connection refused
[18:06:58] <apergos>	 I only get the OK pages
[18:07:00] <apergos>	 kinda useless
[18:07:02] <andrewbogott>	 hashar:  I assume that Tim just forgot to do the production merge for that patch… any reason to think that he'd want to hold it back?
[18:07:11] <LeslieCarr>	 that's weird...
[18:07:18] <hashar>	 andrewbogott: regarding the commits that are pending on sock puppet,  the one you wrote about basically add a  misc::beta::scripts puppet class
[18:07:23] <hashar>	 andrewbogott: not going to cause any harm
[18:07:52] <andrewbogott>	 hashar:  OK, I'll go ahead and merge then.
[18:07:56] <hashar>	 before that there is 7d1927e - "run mergeMessageFileList.php as 'mwdeploy' user"   which might be an issue for code deployment
[18:08:07] <hashar>	 andrewbogott: but should be fine. I tested it on labs
[18:08:39] <hashar>	 once merged, we can verify everything is fine by updating the l10n cache on fenari.  Will give it a poke after dinner
[18:08:39] <andrewbogott>	 Hm, looks like peter beat me to it anyway.
[18:08:45] <hashar>	 ha :-]
[18:09:48] <hashar>	 andrewbogott: dinner time
[18:09:56] <andrewbogott>	 ok.   Thanks for checking in
[18:10:12] <hashar>	 yw ;]
[18:13:59] <apergos>	 I chose a random renderer and looked at atop logs, nothing useful whatsoever...
[18:21:32] <nagios-wm_>	 RECOVERY - NTP on mw61 is OK: NTP OK: Offset -0.01017403603 secs
[18:27:50] <nagios-wm_>	 RECOVERY - Apache HTTP on mw61 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.039 second response time
[18:49:44] <nagios-wm_>	 PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours
[19:10:17] <nagios-wm_>	 PROBLEM - Host db42 is DOWN: PING CRITICAL - Packet loss = 100%
[19:24:21] <Eloquence>	 apergos, I've got a large-ish file upload pending (400 megs), is that good to go or should I wait?
[19:24:49] <apergos>	 seems ok to me
[19:25:03] <apergos>	 all writes go to swift and the netapp now, so I;d say give it a try
[19:25:14] <Eloquence>	 will do
[19:28:52] <gerrit-wm>	 New patchset: Jgreen; "temporary udp log filters for fr-tech to study" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27322
[19:29:46] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27322
[19:36:41] <Jeff_Green>	 !log added a couple temporary filters to oxygen for fr-tech
[19:36:52] <morebots>	 Logged the message, Master
[20:02:03] <gerrit-wm>	 New patchset: awjrichards; "Enable wgMFEnableDesktopResources everywhere" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27325
[20:02:49] <gerrit-wm>	 Change merged: preilly; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27325
[20:12:41] <nagios-wm_>	 PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours
[20:28:53] <nagios-wm_>	 PROBLEM - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is CRITICAL: HTTP CRITICAL - No data received from host
[20:30:32] <nagios-wm_>	 RECOVERY - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is OK: HTTP OK HTTP/1.1 200 OK - 59944 bytes in 0.176 seconds
[20:31:43] <mutante>	 !log authdns-update on dobson, adding fdc.wikimedia.org DNS entry
[20:31:54] <morebots>	 Logged the message, Master
[20:47:56] <LeslieCarr>	 anyone else checking out rendering ?
[20:55:19] <Reedy>	 mutante: who's creating hte fdc wiki?
[20:56:12] <Reedy>	 I'm guessing me as someone CC'd me? :D
[20:58:03] <mutante>	 Reedy: eh:) i wanted to inform you the DNS is now there, since i figured that is the first step in that ticket.. i did not really know the next step yet
[20:59:02] <mutante>	 Reedy: the wiki is: reason: "Per Anasuya's request"
[21:05:10] <LeslieCarr>	 so, it appears that srv220 loves to apache segfault...
[21:06:01] <TimStarling>	 try stopping it, making sure all children are dead, then starting it again
[21:06:59] <LeslieCarr>	 cool
[21:07:26] <LeslieCarr>	 TimStarling does your swift-foo go deep enough to know why the ms-fe's are trying to do go to /usr/local/apache/common/docroot/www.wikimedia.org/w/thumb_handler.php on the rendering cluster ?
[21:07:31] <LeslieCarr>	 because that file doesn't exist ...
[21:07:39] <LeslieCarr>	 AaronSchulz: ^^ ?
[21:07:46] <gerrit-wm>	 New patchset: Hashar; "install upstart conf for beta autoupdater" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27359
[21:07:57] <LeslieCarr>	 !log restarting apache2 on srv220
[21:08:08] <morebots>	 Logged the message, Mistress of the network gear.
[21:08:47] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/27359
[21:12:28] <hashar>	 can someone review/merge https://gerrit.wikimedia.org/r/27359 please ?  I forgot to copy a file from puppet to some directory :-]  That is the upstart configuration file, render the service unstartable :)
[21:15:54] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27359
[21:16:01] <hashar>	 my hero :-)
[21:16:03] <mutante>	 hashar: there you go
[21:16:05] <mutante>	 np
[21:16:07] <hashar>	 thx!
[21:16:40] <hashar>	 mutante: I did some work on the wikibugs script on labs.  Not sure it is ready yet though
[21:17:22] <hashar>	 mutante: as you said,  git::clone{ branch => 'some sha1' }  did not work. So I had to hack git::clone to support a commit sha1
[21:17:58] <mutante>	 hashar: i saw some new patch set that said you can now clone a specific commit id, yea
[21:18:37] <mutante>	 so that worked on labs?
[21:19:06] <hashar>	 yup
[21:19:11] <hashar>	 though that is hacky
[21:19:33] <hashar>	 not sure if   ensure => 'some sha1' is actually wanted. Need to talk about it with paravoid I guess
[21:19:54] <mutante>	 yea, agree
[21:22:07] <hashar>	 feel free to review it though :-]
[21:29:35] <cmjohnson1>	 !log swapping disk 12 out on storage 3
[21:29:47] <morebots>	 Logged the message, Master
[21:32:39] <LeslieCarr>	 AaronSchulz: what exactly should i do ?
[21:35:47] <gerrit-wm>	 New patchset: Dzahn; "add CSS for colored HTTP status, fix logo size" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/27361
[21:36:41] <nagios-wm_>	 PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours
[21:36:41] <nagios-wm_>	 PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours
[21:37:00] <gerrit-wm>	 New patchset: Jgreen; "removing temporary filters from oxygen" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27362
[21:38:00] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27362
[21:38:10] <gerrit-wm>	 Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/27361
[21:41:44] <AaronSchulz>	 LeslieCarr: well /usr/local/apache/common/docroot/www.wikimedia.org/w/thumb_handler.php is a bogus path, but the actual file is there in live-1.5/thumb_handler.php
[21:42:00] <AaronSchulz>	 how often do this errors occur?
[21:42:36] <LeslieCarr>	 about 5-10 times an hour
[21:42:44] <LeslieCarr>	 all from ms-fe's
[21:44:14] <AaronSchulz>	 and the error is on all of the scalars?
[21:44:37] <LeslieCarr>	 yep
[21:44:58] <LeslieCarr>	 oh that's 5-10 per hour per imagescaler
[21:45:26] <LeslieCarr>	 well all of them behind the rendering VIP
[21:46:45] <AaronSchulz>	 LeslieCarr: do you know what the requested url (not fs path) was?
[21:46:59] <AaronSchulz>	 should be like rendering.svc.pmtpa.wmnet/w/thumb_handler.php
[21:47:46] <LeslieCarr>	 not sure
[21:50:47] <LeslieCarr>	 AaronSchulz: yep, not able to figure out how to find out :(
[21:51:31] <hashar>	 Is anyone familiar with Ubuntu Upstart please ?  Igot  a job defined in /etc/init/foobar.conf  but  `start foobar` just says it is unknown.  I got no log in  /var/log/{daemon.log,debug,syslog}  :-(
[21:53:38] <nagios-wm_>	 PROBLEM - Puppet freshness on sq69 is CRITICAL: Puppet has not run in the last 10 hours
[21:53:44] <logmsgbot_>	 !log reedy synchronized php-1.21wmf1/extensions/Collection
[21:53:55] <morebots>	 Logged the message, Master
[22:00:34] <hashar>	 so for the record :   unknown job:  means there is a syntax error somewhere and our upstart version on Lucid does not have any config check command :-(
[22:01:44] <nagios-wm_>	 PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100%
[22:04:35] <nagios-wm_>	 RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms
[22:05:20] <gerrit-wm>	 New patchset: Pyoungmeister; "re-adding mw61 to bits backends" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27366
[22:06:53] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/27366
[22:08:18] <gerrit-wm>	 Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27366
[22:10:29] <nagios-wm_>	 ACKNOWLEDGEMENT - Host db42 is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn hardware failure, Chris is on it, Analytics knows..
[22:10:35] <notpeter>	 !log readded mw61 to bits backends
[22:10:46] <morebots>	 Logged the message, notpeter
[22:21:14] <gerrit-wm>	 New patchset: Dzahn; "adding Apache config for fdc.wikimedia.org (RT-3670)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/27367
[22:21:16] <mutante>	 !log creating new doc root for fdc wiki /usr/local/apache/common/docroot/fdc
[22:21:27] <morebots>	 Logged the message, Master
[22:26:56] <logmsgbot_>	 !log reedy synchronized php-1.21wmf1/resources/jquery/jquery.js  'jQuery 1.8.2'
[22:27:07] <morebots>	 Logged the message, Master
[22:28:46] <gerrit-wm>	 New patchset: Hashar; "beta autoupdater requires git-core" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27371
[22:29:42] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/27371
[22:30:28] <gerrit-wm>	 New patchset: Hashar; "minor typo in wmf autoupdater" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27372
[22:30:50] <hashar>	 two more easy changes :-]
[22:31:26] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/27372
[22:33:07] <gerrit-wm>	 New patchset: awjrichards; "Update path to wgMFCustomLogos" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27373
[22:34:35] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27373
[22:36:24] <hashar>	 off to bed for real now
[22:36:26] <hashar>	 see you
[22:38:32] <logmsgbot_>	 !log reedy synchronized php-1.21wmf1/resources/startup.js  'touch'
[22:38:43] <morebots>	 Logged the message, Master
[22:41:36] <logmsgbot_>	 !log dzahn synchronized /docroot/fdc
[22:41:48] <morebots>	 Logged the message, Master
[22:44:34] <gerrit-wm>	 Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/27367
[22:45:18] <Reedy>	 gerrit died
[22:45:28] <mutante>	 ack, but its temp.
[22:45:34] <mutante>	 i repeated and it worked
[22:50:30] <logmsgbot_>	 !log dzahn synchronized docroot/fdc/index.php
[22:50:42] <morebots>	 Logged the message, Master
[22:50:42] <logmsgbot_>	 dzahn is doing a graceful restart of all apaches
[22:51:03] <logmsgbot_>	 !log dzahn gracefulled all apaches
[22:51:14] <morebots>	 Logged the message, Master
[23:06:02] <logmsgbot_>	 !log dzahn synchronized docroot/fdc/w
[23:06:13] <morebots>	 Logged the message, Master
[23:06:24] <mutante>	 Reedy: https://fdc.wikimedia.org/
[23:08:01] <Reedy>	 mutante: thanks
[23:08:37] <mutante>	 yw, so you got the docroot and ./w/ below it synced
[23:11:49] <logmsgbot_>	 !log awjrichards synchronized wmf-config/InitialiseSettings.php  'Enable sticky cookie fix on all wikis; Update logo asset path for mobile footer'
[23:12:00] <morebots>	 Logged the message, Master
[23:17:37] <logmsgbot_>	 !log awjrichards synchronized php-1.21wmf1/extensions/MobileFrontend/
[23:17:48] <morebots>	 Logged the message, Master
[23:17:52] <awjr>	 binasher, mutante, LeslieCarr - can one of you please purge the mobile varnish cache? we are actively working on getting resourceloader working for MobileFrontend which should mitigate this annoying step!
[23:18:32] <logmsgbot_>	 !log reedy synchronized wmf-config/InitialiseSettings.php  'Add outreachwiki to import sources of enwiki'
[23:18:33] <LeslieCarr>	 awjr: i feel like we have an abusive relationship
[23:18:38] <Krinkle>	 Reedy: Today I noticed my local master-wikis failings due to schema changes that didn't gracefully fallback. So I added stuff to http://wikitech.wikimedia.org/submit/Schema_changes
[23:18:43] <morebots>	 Logged the message, Master
[23:18:45] <awjr>	 LeslieCarr: if i were there i'd pour you a scotch
[23:18:58] <Krinkle>	 I was surprised they're weren't on there yet, need to be careful with the next branchpoint
[23:18:58] <LeslieCarr>	 you always promise there won't be a next time
[23:19:03] <LeslieCarr>	 but there's always a next time
[23:19:21] <Krinkle>	 http://wikitech.wikimedia.org/view/Schema_changes#Pending
[23:19:23] <awjr>	 ha! i never promised such a thing!
[23:19:30] <awjr>	 just that some day there will not be a next time
[23:19:37] <awjr>	 ...
[23:20:05] <Reedy>	 Krinkle: BLOCKING 1.21wmf2 isn't correct
[23:20:19] <Krinkle>	 Reedy: Its not?
[23:20:22] <Krinkle>	 What is it blocking then
[23:20:25] <Reedy>	 We have a global to disable it, which I've already done pre-emptively at the bottom of CommonSettings.php
[23:20:36] <Krinkle>	 I see
[23:20:42] <Reedy>	 It needs to happen "soon", but not before 1.21wmf2
[23:20:47] <mutante>	 !log flushing mobile varnish cache
[23:20:57] <mutante>	 awjr: ^
[23:20:58] <morebots>	 Logged the message, Master
[23:21:02] <Reedy>	 When Asher has done them all, we can then remove the global value change and get the "features"
[23:21:13] <Reedy>	 Krinkle: $wgContentHandlerUseDB
[23:21:32] <awjr>	 thank you mutante :)
[23:21:54] <mutante>	 yw
[23:22:02] <mutante>	 !mobile-cache is http://wikitech.wikimedia.org/view/MobileFrontend#Flushing_the_cache
[23:22:03] <wm-bot>	 Key was added
[23:23:27] <Reedy>	 Krinkle: As for SHA1 hash truncation, it's probably not worth the overhead of dropping and re-creating the indexes (defer to Asher)
[23:23:44] <Reedy>	 rc_moved_to_title, cat_hidden and ss_admins can stay, they're not doing any harm...
[23:25:17] <Krinkle>	 I didn't put a priority on it, just putting it up there. Nuking junk is always good, right? Whenever there is time of course, low priority
[23:26:17] <gerrit-wm>	 New patchset: Tim Starling; "Debugging hack for bug 38273" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27378
[23:26:44] <Reedy>	 I think there's numerous other artefacts still around in the live schemas
[23:27:38] <gerrit-wm>	 Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27378
[23:28:20] <logmsgbot_>	 !log tstarling synchronized wmf-config/InitialiseSettings.php
[23:28:31] <morebots>	 Logged the message, Master
[23:29:29] <TimStarling>	 is there any way to tell what changes are going to be pulled in by a git pull?
[23:29:59] <TimStarling>	 I tried the way that ops do it for puppet, but it doesn't work, it shows a big diff
[23:30:01] <logmsgbot_>	 !log reedy synchronized php-1.20wmf12/resources/jquery/jquery.js  '1.8.2'
[23:30:13] <morebots>	 Logged the message, Master
[23:30:38] <logmsgbot_>	 !log reedy synchronized php-1.20wmf12/resources/startup.js  'touch'
[23:30:49] <morebots>	 Logged the message, Master
[23:31:57] <logmsgbot_>	 !log tstarling synchronized php-1.20wmf12/includes/parser/ParserCache.php
[23:32:08] <morebots>	 Logged the message, Master
[23:32:39] <logmsgbot_>	 !log tstarling synchronized php-1.21wmf1/includes/parser/ParserCache.php
[23:32:50] <morebots>	 Logged the message, Master
[23:34:05] <logmsgbot_>	 !log olivneh synchronized php-1.21wmf1/extensions/E3Experiments
[23:34:16] <morebots>	 Logged the message, Master
[23:35:45] <Ryan_Lane>	 TimStarling: the big diff is what's going to merge
[23:36:36] <Ryan_Lane>	 though it depends on what you diff against, I guess
[23:36:50] <TimStarling>	 git diff HEAD..FETCH_HEAD
[23:37:24] <TimStarling>	 [2337][tstarling@fenari:/home/wikipedia/common/php-1.20wmf12]$ git diff HEAD..FETCH_HEAD | wc -l
[23:37:24] <TimStarling>	 1312695
[23:37:33] <TimStarling>	 1.3 million lines of diff
[23:38:06] <TimStarling>	 and I just ran git pull a few minutes ago
[23:44:30] <mutante>	 TimStarling: is this a difference between 2 or 3 dots? git diff HEAD...FETCH_HEAD | wc -l is just 927
[23:47:31] <TimStarling>	 with 3 dots it still has a lot of weird stuff in it
[23:47:58] <TimStarling>	 it's still nothing like what would happen if you ran git pull
[23:53:12] <binasher>	 Reedy / Krinkle: I thought we possibly were never going to add the contenthandler columns
[23:53:33] <Reedy>	 Oh..
[23:53:55] <binasher>	 if wgContentHandlerUseDB is false, multiple content types are still possible per namespace via title parsing
[23:55:12] <binasher>	 daniel kinzler thought we may want to add them to commons sooner than later, but possibly not elsewhere
[23:56:04] <mutante>	 binasher: fyi. db42 broke. Dario and analytics know and Chris is checking for warranty info
[23:56:29] <binasher>	 Krinkle: are the rc_moved_to_title and ss_admins columns droppable immediately, or only after 1.21wm2 is deployed?
[23:56:40] <Krinkle>	 I wouldn't know
[23:57:20] <Krinkle>	 It certainly doesn't hurt to wait until after, afaik there is no code that requires them to be gone.
[23:57:53] <Reedy>	 rc_moved_to_ns/rc_moved_to_title are used in code
[23:57:58] <Reedy>	 ss_admins wasn't
[23:58:18] <Reedy>	 It was merged in to 1.21wmf1, so after the last migrations tomorrow, all wikis can be cleansed
[23:58:22] <binasher>	 ok.. please note that when adding migrations to the list
[23:58:28] <binasher>	 i might just do them
[23:59:17] <Krinkle>	 I did link to a gerrit change (from which it can be extracted in which branches it is), but I'll keep in mind for next time. Protocol is protocol :)
[23:59:33] <Reedy>	 The changes should trivially merge into 1.20wmf12
[23:59:38] <vvv>	 Where do I report a deployment blocker?