[00:04:18] (03PS1) 10Reedy: Remove nasty old redirect code for stupid print ads [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85374 [00:09:37] !log reedy synchronized w [00:09:41] Logged the message, Master [00:13:04] !log reedy synchronized w [00:14:56] (03PS1) 10Reedy: Remove old loader code for old PHP versions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85378 [00:23:39] !log reedy synchronized w [00:30:17] (03PS1) 10Reedy: 404 page is wrong on sister projects when path starts with /w/ [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85380 [00:30:54] (03CR) 10Reedy: [C: 032] 404 page is wrong on sister projects when path starts with /w/ [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85380 (owner: 10Reedy) [00:31:05] (03CR) 10jenkins-bot: [V: 04-1] 404 page is wrong on sister projects when path starts with /w/ [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85380 (owner: 10Reedy) [00:31:30] (03CR) 10Reedy: "recheck" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85380 (owner: 10Reedy) [00:33:03] (03CR) 10Reedy: [V: 032] 404 page is wrong on sister projects when path starts with /w/ [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85380 (owner: 10Reedy) [00:33:43] (03PS2) 10Reedy: Remove old loader code for old PHP versions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85378 [00:33:48] (03CR) 10Reedy: [C: 032] Remove old loader code for old PHP versions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85378 (owner: 10Reedy) [00:33:59] (03Merged) 10jenkins-bot: Remove old loader code for old PHP versions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85378 (owner: 10Reedy) [00:36:15] (03PS2) 10Reedy: Remove nasty old redirect code for stupid print ads [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85374 [00:36:21] (03CR) 10Reedy: [C: 032] Remove nasty old redirect code for stupid print ads [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85374 (owner: 10Reedy) [00:36:32] (03Merged) 10jenkins-bot: Remove nasty old redirect code for stupid print ads [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85374 (owner: 10Reedy) [00:38:42] !log reedy synchronized docroot and w [00:38:46] Logged the message, Master [00:40:39] !log reedy synchronized docroot and w [00:41:37] !log reedy synchronized docroot and w [00:43:54] !log reedy synchronized docroot and w [00:45:07] !log reedy synchronized docroot and w [00:46:24] (03PS1) 10Reedy: Add $prot back in [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85385 [00:47:06] (03CR) 10Reedy: [C: 032] Add $prot back in [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85385 (owner: 10Reedy) [00:47:15] (03Merged) 10jenkins-bot: Add $prot back in [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85385 (owner: 10Reedy) [00:51:33] (03PS1) 10Springle: roll out mariadb to db1044 [operations/puppet] - 10https://gerrit.wikimedia.org/r/85386 [00:51:59] (03PS1) 10Reedy: Temporarily lift account creation limit for an IP for English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85387 [00:52:24] (03CR) 10Reedy: [C: 032] Temporarily lift account creation limit for an IP for English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85387 (owner: 10Reedy) [00:52:30] (03CR) 10Springle: [C: 032] roll out mariadb to db1044 [operations/puppet] - 10https://gerrit.wikimedia.org/r/85386 (owner: 10Springle) [00:52:34] (03Merged) 10jenkins-bot: Temporarily lift account creation limit for an IP for English Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85387 (owner: 10Reedy) [00:53:17] !log reedy synchronized wmf-config/throttle.php [00:53:20] Logged the message, Master [01:02:40] (03CR) 10Helder.wiki: "LOL" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85374 (owner: 10Reedy) [01:11:31] (03PS1) 10Springle: db1044 prefer mariadb take 2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/85390 [01:12:32] (03CR) 10Springle: [C: 032] db1044 prefer mariadb take 2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/85390 (owner: 10Springle) [01:16:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:17:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 15.444 second response time [01:23:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:25:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 11.811 second response time [01:29:12] (03PS1) 10Springle: manual mariadb 10 beta test [operations/puppet] - 10https://gerrit.wikimedia.org/r/85392 [01:31:22] (03CR) 10Springle: [C: 032] manual mariadb 10 beta test [operations/puppet] - 10https://gerrit.wikimedia.org/r/85392 (owner: 10Springle) [01:32:37] PROBLEM - HTTP on formey is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:27] RECOVERY - HTTP on formey is OK: HTTP OK: HTTP/1.1 200 OK - 3596 bytes in 0.054 second response time [01:34:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:37:16] (03PS1) 10Reedy: Fixup whitespace [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85393 [01:37:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.219 second response time [01:37:53] (03CR) 10Reedy: [C: 032] Fixup whitespace [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85393 (owner: 10Reedy) [01:38:02] (03Merged) 10jenkins-bot: Fixup whitespace [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85393 (owner: 10Reedy) [01:38:42] !log reedy synchronized docroot and w [01:40:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:40:28] (03PS1) 10Reedy: Remove mobileRedirect.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85394 [01:48:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 20.679 second response time [02:01:45] !log LocalisationUpdate completed (1.22wmf17) at Sat Sep 21 02:01:44 UTC 2013 [02:01:49] Logged the message, Master [02:02:26] !log LocalisationUpdate completed (1.22wmf18) at Sat Sep 21 02:02:26 UTC 2013 [02:02:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [02:02:30] Logged the message, Master [02:05:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 19.729 second response time [02:07:37] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Sep 21 02:07:36 UTC 2013 [02:07:40] Logged the message, Master [02:15:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [02:16:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 17.733 second response time [02:34:56] (03PS8) 10Reedy: Throttle now handles IP ranges. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/65644 (owner: 10Dereckson) [02:46:46] (03CR) 10Reedy: [C: 031] deployment: abstract out MW_RSYNC_HOST [operations/puppet] - 10https://gerrit.wikimedia.org/r/72491 (owner: 10Hashar) [03:07:08] !log imagescalers in eqiad shows a drop of CPU/network at around 1:40am UTC. Thumbnails are no more rendered :-( [03:07:11] Logged the message, Master [03:08:35] hashar: shouldn't you still be asleep? [03:08:44] bd808: I should [03:08:53] but wanted to happen the cocktails party in the office [03:09:11] Good plan. :) [03:09:11] I opened an eye at 4am feeling fin [03:09:17] kissed my wife, and went up [03:09:29] to find out the cluster is no more generating thumbnails hehe [03:09:42] I'm hoping that my flight eventually leaves SFO [03:10:13] bd808: last week mine got delayed by 1 hour and half :/ [03:10:17] Damn thumbnails. I only somebody was working on that. *grin* [03:10:24] but luckily it managed to sprint back to France I have caught my connection! [03:10:53] well the thumbnails infrastructure is a bit scary, there are so many layers involved that I am still surprised how stable it is [03:11:29] I'm drinking beer and watching American football so things could be worse. [03:12:31] yeah you could be attending the football match with no beer [03:15:26] both me and Reedy are looking at it [03:15:52] our hero! sorry to abort your friday night social time :-( [03:16:49] !log reedy synchronized docroot and w [03:17:08] Reedy: \O/ [03:17:11] got thumbnails [03:18:24] what ever happened, it should probably errors out somehow whenever it happens [03:18:42] yep, fixed [03:18:49] thumb.php vs thumb_handler.php [03:19:18] no idea what it means [03:19:29] define( 'THUMB_HANDLER', true ); [03:20:24] (03PS1) 10Reedy: Revert changes to thumb_handler, leave pointing at itself [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85397 [03:20:41] (03CR) 10Reedy: [C: 032] Revert changes to thumb_handler, leave pointing at itself [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85397 (owner: 10Reedy) [03:20:56] (03Merged) 10jenkins-bot: Revert changes to thumb_handler, leave pointing at itself [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85397 (owner: 10Reedy) [03:21:04] ahh [03:21:05] !log Scalers fixed [03:21:09] Logged the message, Master [03:21:32] thumb_handler.php being an old entry point we kept around for compatibility ? [03:21:50] if ( defined( 'THUMB_HANDLER' ) ) { [03:21:51] // Called from thumb_handler.php via 404; extract params from the URI... [03:21:51] wfThumbHandle404(); [03:21:51] } else { [03:21:51] // Called directly, use $_GET params [03:21:53] ah no damn heeheh [03:21:53] wfThumbHandleRequest(); [03:21:55] } [03:21:56] crazy rooting [03:22:00] routing [03:22:12] talk about hard to understand/follow code paths [03:22:56] * hashar escapes [03:23:07] paravoid: Reedy: thanks and enjoy the social time! [03:24:38] Reedy: I think you can blame AaronSchulz for that interesting stub file. [03:25:40] haha [03:25:54] rage. [03:26:44] bd808: moaar tech debt [03:27:29] I have job security; at least as soon as I figure out how to fix things. [03:28:52] and after you crashed the cluster once [03:29:39] Working on that. My first core patch deployed yesterday. Should hit enwiki in a week I guess. [03:30:10] \O/ [03:32:05] It's a tiny change but for a bug Reedy filed, so I'm helping the team! [03:32:17] I file ALL of the bugs [03:32:18] ;) [03:32:48] Because you read the logs. [03:34:01] Which reminds me I need to ask for some more rights next week. paravoid scoffed when I said I didn't have prod access. [03:36:39] Phone battery dying. Talk to you guys another day. [06:26:14] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [06:26:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.245 second response time [07:00:34] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [07:01:24] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [07:03:34] PROBLEM - Apache HTTP on mw1085 is CRITICAL: Connection refused [07:09:34] RECOVERY - Apache HTTP on mw1085 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.091 second response time [07:10:58] (03CR) 10MaxSem: [C: 031] "Ugh, I thought it died a horrible death last year. Die zombie mobileRedirect.php die." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85394 (owner: 10Reedy) [07:16:04] PROBLEM - NTP on mw1085 is CRITICAL: NTP CRITICAL: Offset unknown [07:21:04] RECOVERY - NTP on mw1085 is OK: NTP OK: Offset -0.003455519676 secs [08:23:43] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [08:25:13] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.54 ms [11:06:33] (03PS1) 10TTO: Category collation for viwikivoyage to uca-vi [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85419 [11:15:44] (03PS2) 10TTO: Category collation for viwikivoyage to uca-vi [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85419 [12:07:29] PROBLEM - SSH on rubidium is CRITICAL: Server answer: [12:09:29] RECOVERY - SSH on rubidium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [14:31:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [14:32:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [14:52:48] Could you abandon the following change? https://gerrit.wikimedia.org/r/#/c/84203/ Superseded by Ia4ddd7fa5b087ec08a1a7f94b48d20998bb288ad [14:53:27] (03CR) 10Dereckson: "Superseded by Ia4ddd7fa5b087ec08a1a7f94b48d20998bb288ad" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84203 (owner: 10Ladsgroup) [14:53:45] Oh, it's done. Already abanoned by code submitter, perfect; [16:22:44] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:24:34] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [17:18:43] (03CR) 10Ottomata: "(1 comment)" [operations/debs/kafka] (debian) - 10https://gerrit.wikimedia.org/r/85219 (owner: 10Ottomata) [18:42:34] !log Jenkins: upgrading pep8 from 1.4.5 to 1.4.6 on gallium and lanthanum [18:42:37] Logged the message, Master [23:50:40] (03PS1) 10TTO: Allow crats on outreachwiki to revoke translationadmin group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85513