[00:08:06] (03PS11) 10Ori.livneh: Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis) [00:08:15] (03CR) 10Ori.livneh: [C: 032 V: 032] Logstash puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/100395 (owner: 10BryanDavis) [00:08:54] \o/ [00:17:01] (03PS1) 10Ori.livneh: logstash: re-order ::redis::ganglia include to avoid resource conflict [operations/puppet] - 10https://gerrit.wikimedia.org/r/105870 [00:17:59] (03CR) 10Ori.livneh: [C: 032 V: 032] logstash: re-order ::redis::ganglia include to avoid resource conflict [operations/puppet] - 10https://gerrit.wikimedia.org/r/105870 (owner: 10Ori.livneh) [00:25:37] (03PS1) 10Ori.livneh: logstash: Comment out openjdk-7-jdk package resource [operations/puppet] - 10https://gerrit.wikimedia.org/r/105872 [00:26:15] (03CR) 10Ori.livneh: [C: 032 V: 032] logstash: Comment out openjdk-7-jdk package resource [operations/puppet] - 10https://gerrit.wikimedia.org/r/105872 (owner: 10Ori.livneh) [00:29:48] (03PS1) 10Ori.livneh: logstash: rename elasticsearch.conf.erb -> elasticsearch.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/105874 [00:30:05] (03CR) 10Ori.livneh: [C: 032 V: 032] logstash: rename elasticsearch.conf.erb -> elasticsearch.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/105874 (owner: 10Ori.livneh) [00:43:10] (03PS1) 10BryanDavis: Fix circular dependency in Logstash::Conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/105876 [00:43:19] ori: ^ [00:45:13] (03CR) 10Ori.livneh: [C: 04-1] "You missed a few:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105876 (owner: 10BryanDavis) [00:45:25] bd808: ^ [00:45:36] grr. Thanks [00:47:30] (03PS2) 10BryanDavis: Fix circular dependency in Logstash::Conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/105876 [00:49:28] (03CR) 10Ori.livneh: [C: 032] Fix circular dependency in Logstash::Conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/105876 (owner: 10BryanDavis) [00:56:20] !log aaron started scap: timing test (beta) [00:56:39] Logged the message, Master [01:01:54] (03PS1) 10BryanDavis: Remove unnecessary reference to logstash class [operations/puppet] - 10https://gerrit.wikimedia.org/r/105877 [01:02:44] ori: ^ Another attempt [01:03:59] (03CR) 10Ori.livneh: [C: 032] Remove unnecessary reference to logstash class [operations/puppet] - 10https://gerrit.wikimedia.org/r/105877 (owner: 10BryanDavis) [01:16:35] (03PS1) 10BryanDavis: Fix puppet file source declarations for logstash [operations/puppet] - 10https://gerrit.wikimedia.org/r/105878 [01:17:21] (03CR) 10Ori.livneh: [C: 032 V: 032] Fix puppet file source declarations for logstash [operations/puppet] - 10https://gerrit.wikimedia.org/r/105878 (owner: 10BryanDavis) [01:19:36] !log aaron finished scap: timing test (beta) (duration: 25m 28s) [01:19:55] Logged the message, Master [01:30:33] (03PS6) 10BryanDavis: Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 [01:30:42] (03CR) 10jenkins-bot: [V: 04-1] Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis) [01:33:31] !log aaron started scap: timing test (beta) [01:33:41] (03PS7) 10BryanDavis: Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 [01:33:47] Logged the message, Master [01:37:59] !log aaron finished scap: timing test (beta) (duration: 04m 39s) [01:38:16] Logged the message, Master [01:39:06] (03CR) 10BryanDavis: "Patch set 7 was a manual rebase." [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis) [01:39:07] !log aaron started scap: timing test (beta) [01:39:23] Logged the message, Master [01:41:29] !log aaron finished scap: timing test (beta) (duration: 02m 28s) [01:41:45] Logged the message, Master [01:57:03] (03PS1) 10Aaron Schulz: Fixed "failed to open dir" errors with single version --versions [operations/puppet] - 10https://gerrit.wikimedia.org/r/105884 [02:21:54] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:03] !log LocalisationUpdate completed (1.23wmf8) at Tue Jan 7 02:22:03 UTC 2014 [02:26:24] PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:44] RIP morebots [02:40:29] andrewbogott: Can you fix? [02:40:53] only briefly, it seems [02:41:13] Heh. [02:41:30] !log LocalisationUpdate completed (1.23wmf9) at Tue Jan 7 02:41:30 UTC 2014 [02:42:02] morebots: wb [02:42:02] I am a logbot running on tools-exec-03. [02:42:02] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [02:42:02] To log a message, type !log . [02:42:09] nothing in the error log :( [02:43:48] Heh, error !log. [02:43:54] I amuse myself. [02:52:34] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [02:53:04] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.38 ms [02:55:14] PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused [02:56:14] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.426 second response time [03:01:14] RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 0.123 second response time [03:01:54] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.990 second response time [03:16:06] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 7 03:16:06 UTC 2014 [03:16:25] Logged the message, Master [03:18:03] !log springle synchronized wmf-config/db-eqiad.php 'warm up db1006' [03:18:21] Logged the message, Master [03:21:54] "PHP fatal error in /usr/local/apache/common-local/php-1.23wmf8/extensions/GlobalBlocking/SpecialGlobalBlock.php line 278: [03:21:54] Call to a member function getPrefixedText() on a non-object " [03:22:08] active spambot [03:22:11] pretty urgent [03:22:12] yes [03:22:23] Tried again, same error [03:26:44] PROBLEM - RAID on ms-be11 is CRITICAL: CRITICAL: 1 failed LD(s) (Offline) [03:26:45] PROBLEM - Disk space on ms-be11 is CRITICAL: DISK CRITICAL - /srv/swift-storage/sde1 is not accessible: Input/output error [03:28:48] (03CR) 10KartikMistry: [C: 031] "LGTM too from debian/ changes." [operations/debs/vips] - 10https://gerrit.wikimedia.org/r/102617 (owner: 10coren) [03:29:05] Bsadowski1: like https://gerrit.wikimedia.org/r/#/c/103542/ I guess [03:30:40] Maybe I entered a field in wrong (like an extra character) [03:30:40] it's in wmf9 but not wmf8 [03:30:47] Ah [03:32:57] https://noc.wikimedia.org/conf/wikiversions.dat.txt [03:33:59] Ah cool. Thanks Mzm [03:34:56] TimStarling: Hmmm, maybe we should just redirect noc to Meta-Wiki. [03:35:24] Why? [03:35:26] We could have a similar index of useful files on a wiki page. [03:35:31] Hm [03:35:41] We'd previously discussed getting rid of conf/. [03:35:46] In the MediaWiki namespace perhaps? [03:35:51] Or reëngineering it. [03:36:02] [[m:Wikimedia configuration]] [03:36:04] Or something. [03:36:19] reëngineering? [03:36:50] Reïmplementing? [03:36:55] Redoing? [03:37:06] Rejigging. [03:37:38] Bsadowski1: noc.wm.o/conf made a lot more sense before the configuration files were in Git. [03:37:47] It used to be the only way to see them if you didn't have a shell account. [03:38:07] And they used to be periodically synced. It wasn't great. [03:38:26] Hmm, I guess a bot could sync it to the wiki. [03:38:45] You could just have links. [03:38:50] We already have several Git repo viewers. [03:39:25] Krinkle|detached: Thoughts? ^^ [03:41:20] cherry picking a change to an extension is ridiculously complicated [03:41:26] does someone have a script for this or something? [03:41:45] I'm about 2/3 of the way through it [03:41:48] TimStarling: What do you mean by cherry picking to an extension exactly? [03:41:53] Is it like a cross-repo cherry-pick? [03:42:11] RoanKattouw: Putting https://gerrit.wikimedia.org/r/#/c/103542/ in wmf8. [03:42:18] I mean like https://gerrit.wikimedia.org/r/#/c/105889/ [03:42:31] I created the branch with the gerrit UI, cherry picked the change into it [03:42:32] OK, yeah [03:42:37] Yes [03:42:41] That's how you do it [03:42:44] now I'm about to update the submodule (I already have a script for that part) [03:43:05] well, a known method anyway: [03:43:08] git update-index --cacheinfo 160000 dd3fe6734e436fa216e5c9ad2c6418c8ea064e77 extensions/GlobalBlocking [03:43:25] There isn't any automation for it other than that you can ask Reedy to automatically create the wmf/* branches for a given extension [03:43:35] VE has that enabled because we cherry-pick things more often than not [03:43:47] But for extensions like GlobalBlocking it's rarely necessary [03:44:04] it should be done more often, if it was easier then people would do it [03:44:14] Probably [03:44:46] For updating the submodule, I just do cd extensions/VisualEditor && git fetch && git checkout origin/wmf/1.23wmf8 && cd ../.. && git commit -a && git review [03:45:37] I should probably script that [03:46:00] I use fetch+checkout so I don't have to copy hashes around and because I often cherry-pick multiple things in one sitting [03:48:36] where are you btw? [03:49:03] at morning tea some radio guy said he wanted to interview you [03:49:22] hah nice [03:49:30] Right now I'm at the accommodation at St George's college [03:49:53] I am in the open programming miniconf [03:49:53] Because there wasn't much interesting stuff going on before lunch (except the keynote, which I did go to) and James and I have slides to write [03:49:57] Nice [03:50:05] I'm going to join the open programming miniconf after lunch [03:50:15] I would be hearing about MariaDB graph stuff if I wasn't too busy backporting a 3-line change [03:50:24] haha [03:50:34] The 13:20 talk in that miniconf is about PHP so I figured I should go [03:52:20] I just got in last night, I'm staying at trinity [03:53:10] I hear it was hot yesterday, nice to see you survived [03:53:20] I know you don't like it when it gets above 18°C [03:54:03] Yeah it was hot [03:54:18] Fortunately I'm getting out of here before it hits 43 C on Saturday [03:54:35] Wait, is Trinity College where gmaps tells me it is? All the way on the other side of the CBD? [03:54:47] no [03:55:24] Oh I see it now [03:55:52] It's where the main drag crosses the main road [03:57:14] TimStarling: So, any way I can locate this radio guy? Hang out in the registration area and try to find him there, or something? [03:57:55] (03PS1) 10Springle: db1006 back to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105891 [03:58:25] (03CR) 10Springle: [C: 032] db1006 back to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105891 (owner: 10Springle) [03:59:18] !log springle synchronized wmf-config/db-eqiad.php 'db1006 full steam' [03:59:34] he walks around and asks people to record interviews during breaks [03:59:36] Logged the message, Master [04:00:02] lunch is uncatered, so probably afternoon tea will be the next time he tries it [04:01:15] ah, nice, someone asked a hard question and Arjen Lentz jumped up on stage to answer it [04:03:26] It's funny, either I missed all the Europeans last time or there are more Europeans this time around [04:03:56] In 2012 I got the feeling I was the only person from Europe there [04:04:08] To be fair, Europe is really far away :) [04:10:17] !log tstarling synchronized php-1.23wmf8/extensions/GlobalBlocking/SpecialGlobalBlock.php [04:10:35] Logged the message, Master [05:11:04] is someone from the echo team here? [05:11:24] https://www.mediawiki.org/wiki/Special:RecentChanges a bunch of spambots hitting Talk:Sandbox [05:11:29] and since it's flow it can't be semi-ed [05:11:51] 1 a minute [05:12:04] oh poop [05:12:47] also, it isn't checking to see if the user is blocked [05:12:50] it's Flow [05:12:53] I just poked StevenW [05:12:59] if only I remembered the names of the people I met [05:13:10] TimStarling's away, ugh [05:29:31] (03PS2) 10Aaron Schulz: Fixed "failed to open dir" errors with single version --versions [operations/puppet] - 10https://gerrit.wikimedia.org/r/105884 [05:30:16] ori: ^ [05:55:08] Hey, what's up? [05:55:31] hi StevenW [05:55:35] (I'm not technically on the Flow team, btw. But I can ping the right people.) [05:55:43] mm, crisis [05:55:47] James_F said we're on it [05:56:32] StevenW: spambots hit flow, and apparently block/abusefilter/semi don't work [05:56:33] StevenW: ori and ebernhardson are dealing. [05:56:38] Cool. [05:56:38] https://gerrit.wikimedia.org/r/#/c/105896/ [05:56:48] ebernhardson's change seems better than oris' [05:57:29] wasn't flow rolled out knowing that none of the spam protection mechanisms worked? [05:58:15] yes, it was based on some stats we ran showing that talk namespace spam is a tiny fraction of overall abuse filter hits [05:58:23] Ryan_Lane: The assumption was that very very few pages (and no ability to add to the list like LQT) plus the built-in delete/hide functions would suffice. [05:58:27] also abuse filter for flow is in gerrit, just not finished yet [05:58:37] heh [05:58:48] doesn't anyone remember poor AFT? [06:00:02] if you present a box to spambots, it'll get abused [06:00:13] I'll abuse your spambox. [06:00:21] \o/ [06:00:28] wait. is that a good thing, or a bad thing? [06:00:40] I don't know. I just couldn't pass up the opportunity. [06:00:43] :D [06:01:01] * YuviPanda thanktrolls StevenW [06:02:16] Anyway, lessons learned. [06:02:25] Clearly not. [06:02:39] going to deploy ebernhardson's temp. workaround [06:03:20] In my opinion, block is the bare minimum that should be supported [06:03:30] everything else can wait [06:03:57] !log ori synchronized php-1.23wmf9/extensions/Flow 'Ic096bfca3: Emergency spam prevention' [06:04:04] ^ ebernhardson [06:04:13] and that, folks, is it for now [06:04:15] Logged the message, Master [06:04:42] now we get a PHP fatal error in /usr/local/apache/common-local/php-1.23wmf9/extensions/Flow/FlowActions.php line 101: [06:04:44] Class 'Container' not found [06:04:56] oops [06:06:03] erik is on it [06:06:17] well /that/'ll stop the spam at least [06:08:31] Oh, other Erik. [06:08:36] I got so confused for a minute. [06:11:36] !log ori synchronized php-1.23wmf9/extensions/Flow 'I00d32ecec: Use Container namespace' [06:11:54] Logged the message, Master [06:12:02] ^ ebernhardson [06:12:04] looks better [06:12:11] ok thanks [06:19:52] !log ebernhardson Flow pages on mw were targeted by a spambot, deployed Ic096bfca35f3da5ebe508e224fecac6361753061 to put flow in read-only mode. [06:20:10] Logged the message, Master [06:35:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:37:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:39:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:41:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:43:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:45:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:47:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:49:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:51:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:53:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:55:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:57:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:59:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:30:29 AM UTC [06:59:56] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Tue Jan 7 06:59:55 UTC 2014 [07:02:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:59:55 AM UTC [07:03:14] (03PS1) 10BBlack: add fallocate patch [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/105901 [07:03:15] (03PS1) 10BBlack: varnish (3.0.3plus~rc1-wm28) precise; urgency=low [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/105902 [07:03:48] (03PS1) 10BBlack: add fallocate patch [operations/debs/varnish] (3.0.5-plus-wm) - 10https://gerrit.wikimedia.org/r/105903 [07:03:49] (03PS1) 10BBlack: Updated netmapper patch to e4eb4160 [operations/debs/varnish] (3.0.5-plus-wm) - 10https://gerrit.wikimedia.org/r/105904 [07:03:50] (03PS1) 10BBlack: varnish (3.0.5plus~wmftest-wm2) unstable; urgency=low [operations/debs/varnish] (3.0.5-plus-wm) - 10https://gerrit.wikimedia.org/r/105905 [07:04:06] PROBLEM - Puppet freshness on mw86 is CRITICAL: Last successful Puppet run was Tue 07 Jan 2014 06:59:55 AM UTC [07:05:34] (03CR) 10BBlack: [C: 032 V: 032] add fallocate patch [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/105901 (owner: 10BBlack) [07:05:50] (03CR) 10BBlack: [C: 032 V: 032] varnish (3.0.3plus~rc1-wm28) precise; urgency=low [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/105902 (owner: 10BBlack) [07:06:12] (03CR) 10BBlack: [C: 032 V: 032] add fallocate patch [operations/debs/varnish] (3.0.5-plus-wm) - 10https://gerrit.wikimedia.org/r/105903 (owner: 10BBlack) [07:06:25] (03CR) 10BBlack: [C: 032 V: 032] Updated netmapper patch to e4eb4160 [operations/debs/varnish] (3.0.5-plus-wm) - 10https://gerrit.wikimedia.org/r/105904 (owner: 10BBlack) [07:06:35] (03CR) 10BBlack: [C: 032 V: 032] varnish (3.0.5plus~wmftest-wm2) unstable; urgency=low [operations/debs/varnish] (3.0.5-plus-wm) - 10https://gerrit.wikimedia.org/r/105905 (owner: 10BBlack) [07:30:41] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Tue Jan 7 07:30:35 UTC 2014 [08:01:01] PROBLEM - MySQL Processlist on db1006 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 73 statistics [08:02:01] RECOVERY - MySQL Processlist on db1006 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 3 statistics [08:29:46] !log nikerabbit synchronized php-1.23wmf9/extensions/UniversalLanguageSelector/resources/css/ext.uls.css 'ULS bigfix' [08:30:04] Logged the message, Master [08:31:13] Nikerabbit: Bigfix or bugfix? :-) [08:31:19] !log nikerabbit synchronized php-1.23wmf9/extensions/UniversalLanguageSelector/lib/jquery.uls/src/jquery.uls.regionfilter.js 'ULS bugfix' [08:31:28] !log logging table partitioning test on s7 db1041 [08:31:37] Logged the message, Master [08:31:55] Logged the message, Master [08:32:34] Guten Tag. I am going to upgrade Zuul this morning so jobs will be disrupted a bit. Will ping whenever I start. [08:42:53] !log nikerabbit started scap: CLDR 24 plural rule backport [08:43:10] Logged the message, Master [08:45:09] hashar: Sorry. :-( [08:47:06] (03CR) 10Ori.livneh: [C: 032] Fixed "failed to open dir" errors with single version --versions [operations/puppet] - 10https://gerrit.wikimedia.org/r/105884 (owner: 10Aaron Schulz) [08:49:21] hashar: Any idea why it only broke on 2 Jan? [08:49:31] hashar: The new repo was created in December… [08:49:45] no idea [08:49:57] Last commit in mediawiki/extensions.git is : [08:49:58] commit 2f1536a400cfc4270cd202324f3473dad251ad73 [08:49:58] Date: Thu Jan 2 09:31:07 2014 +0100 [08:50:29] hold on no [08:50:31] grrr [08:53:59] hashar: Have created https://bugzilla.wikimedia.org/show_bug.cgi?id=59758 for this. [09:14:06] hashar: I wonder if VisualEditor having a submodule might complicate things? [09:16:46] James_F: I have no clue [09:16:46] James_F: I looked on bastion and on a local checkout, I have no idea what might have broken the mediawiki/extensions.git repo [09:16:46] might not be related to VisualEditor [09:16:55] * James_F nods. [09:16:58] Sigh. [09:18:45] !log nikerabbit finished scap: CLDR 24 plural rule backport (duration: 40m 10s) [09:19:01] Logged the message, Master [09:19:11] hurray [09:23:46] so this was the first scap after "Rebuild lolcalisation cache"? (Pun intended in that one? :) ) [09:24:25] !log upgrading Zuul [09:24:42] Logged the message, Master [09:26:03] lolcatization [09:27:50] !log restarting Zuul and enabling Gearman plugin in Jenkins [09:28:07] Logged the message, Master [09:29:01] PROBLEM - zuul_service_running on gallium is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/bin/python /usr/local/bin/zuul-server [09:29:47] ^^^ that is me [09:32:36] !log restarted Zuul with the Gearman backend. Jobs might ends up being broken :( [09:32:52] Logged the message, Master [10:18:46] !log mediawiki-core-lint job is broken on gallium (can't clone repository) [10:19:04] Logged the message, Master [10:30:59] (03PS1) 10Ori.livneh: Move misc::graphite::navtiming into webperf role [operations/puppet] - 10https://gerrit.wikimedia.org/r/105928 [10:31:45] (03CR) 10jenkins-bot: [V: 04-1] Move misc::graphite::navtiming into webperf role [operations/puppet] - 10https://gerrit.wikimedia.org/r/105928 (owner: 10Ori.livneh) [10:33:41] (03PS2) 10Ori.livneh: Move misc::graphite::navtiming into webperf role [operations/puppet] - 10https://gerrit.wikimedia.org/r/105928 [10:35:10] (03CR) 10Ori.livneh: [C: 032] Move misc::graphite::navtiming into webperf role [operations/puppet] - 10https://gerrit.wikimedia.org/r/105928 (owner: 10Ori.livneh) [10:43:46] any smart one knows what happens to stdout/stderr when using start-stop-daemon ? [10:44:04] on gallium I am using git-daemon via an init script, and I have ZERO idea where logs might ends up :( [10:44:25] ori: how about https://gerrit.wikimedia.org/r/#/c/105614/ ? :) [10:44:37] hashar: if there is no other specific target specified it often just ends up in /var/log/syslog [10:44:45] (03CR) 10Ori.livneh: [C: 032 V: 032] Also logbase 2 for the shorter reqerror graphs [operations/puppet] - 10https://gerrit.wikimedia.org/r/105614 (owner: 10Nemo bis) [10:45:14] mutante: you are my hero [10:47:14] Nemo_bis: try now [10:48:50] (03PS1) 10Ori.livneh: Clean up leftover misc::graphite junk [operations/puppet] - 10https://gerrit.wikimedia.org/r/105933 [10:50:54] (03CR) 10Ori.livneh: [C: 032] Clean up leftover misc::graphite junk [operations/puppet] - 10https://gerrit.wikimedia.org/r/105933 (owner: 10Ori.livneh) [10:58:43] ori: preeeetty, but the first doesn't load O_o https://gdash.wikimedia.org/dashboards/reqerror/ [10:59:31] graphite gives error 500 for that one, maybe needs to warm up or something? [11:01:50] hm, odd. no, no warm required. must be something else. [11:12:03] https://www.mediawiki.org/wiki/Project:Current_issues#Flow_is_down.3F_37700 [11:14:06] * twkozlowski murmurs ♪ Flow is down, can't write stuff, 'cause something's gone very bad ♪ [11:18:32] (03Abandoned) 10Hashar: lint applicationserver::cron [operations/puppet] - 10https://gerrit.wikimedia.org/r/100784 (owner: 10Hashar) [11:18:53] hashar: take a glance at RT 6594 and 6593 .. fyi only [11:18:55] (03Abandoned) 10Hashar: beta: get rid of lucene search configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/100771 (owner: 10Hashar) [11:19:07] (03PS3) 10Hashar: contint: package curl on slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/97526 [11:19:35] (03PS2) 10Hashar: stages.pp puppet lint fixes [operations/puppet] - 10https://gerrit.wikimedia.org/r/104919 [11:19:45] (03PS2) 10Hashar: applicationserver: pass puppetlint / retab [operations/puppet] - 10https://gerrit.wikimedia.org/r/104920 [11:20:08] (03PS2) 10Hashar: beta: remove old parsoid updater [operations/puppet] - 10https://gerrit.wikimedia.org/r/104978 [11:20:23] (03Abandoned) 10Hashar: ganglia_new: fix puppet-lint issues [operations/puppet] - 10https://gerrit.wikimedia.org/r/104921 (owner: 10Hashar) [11:21:09] (03CR) 10Hashar: "it is no more used, so change can be merged anytime." [operations/puppet] - 10https://gerrit.wikimedia.org/r/104978 (owner: 10Hashar) [11:23:21] mutante: looking [11:23:31] !rt 6593 [11:23:31] http://rt.wikimedia.org/Ticket/Display.html?id=6593 [11:23:37] !rt 6594 [11:23:37] http://rt.wikimedia.org/Ticket/Display.html?id=6594 [11:26:07] (03CR) 10Dzahn: [C: 031] "replaced by jenkins job = nice! per hashar's comments and https://bugzilla.wikimedia.org/show_bug.cgi?id=57233" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104978 (owner: 10Hashar) [11:34:39] (03CR) 10Dzahn: [C: 032] "yea, makes sense as a requirement for qunit" [operations/puppet] - 10https://gerrit.wikimedia.org/r/97526 (owner: 10Hashar) [11:34:48] thx [11:35:19] hashar: it's on master [11:56:58] (03PS1) 10Hashar: varnish: adds gallium to misc-varnish [operations/puppet] - 10https://gerrit.wikimedia.org/r/105939 [11:58:27] (03CR) 10Hashar: "We will have to switch the DNS later on but this change at least provide the basic setup that would let test out misc varnish works fine f" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105939 (owner: 10Hashar) [11:58:45] (03CR) 10Hashar: "redone on https://gerrit.wikimedia.org/r/#/c/105939/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/82653 (owner: 10Hashar) [12:18:56] twkozlowski: I'm trying to imagine the tune for that [12:58:24] (03PS1) 10Dzahn: do not use generated .htaccess from Bugzilla [operations/puppet] - 10https://gerrit.wikimedia.org/r/105944 [13:00:01] !change 105944 | andre__ [13:00:02] andre__: https://gerrit.wikimedia.org/r/#q,105944,n,z [13:01:08] (03CR) 10Dzahn: "hrmpf, yea, will fix spaces/tabs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105944 (owner: 10Dzahn) [13:01:28] mutante, testing and production refer to zirconium for that patch? [13:01:38] andre__: yes [13:02:53] (03CR) 10Aklapper: [C: 031] "LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105944 (owner: 10Dzahn) [13:04:58] thx,bbl [13:33:53] (03PS1) 10Mark Bergsma: Separate out the config section into misc::smokeping::config [operations/puppet] - 10https://gerrit.wikimedia.org/r/105946 [13:33:54] (03PS1) 10Mark Bergsma: Add web service for smokeping [operations/puppet] - 10https://gerrit.wikimedia.org/r/105947 [13:35:26] (03PS2) 10Mark Bergsma: Separate out the config section into misc::smokeping::config [operations/puppet] - 10https://gerrit.wikimedia.org/r/105946 [13:35:27] (03PS2) 10Mark Bergsma: Add web service for smokeping [operations/puppet] - 10https://gerrit.wikimedia.org/r/105947 [13:37:08] (03CR) 10Mark Bergsma: [C: 032] Separate out the config section into misc::smokeping::config [operations/puppet] - 10https://gerrit.wikimedia.org/r/105946 (owner: 10Mark Bergsma) [13:37:31] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [13:37:56] (03CR) 10Mark Bergsma: [C: 032] Add web service for smokeping [operations/puppet] - 10https://gerrit.wikimedia.org/r/105947 (owner: 10Mark Bergsma) [13:44:58] (03PS1) 10Mark Bergsma: Move smokeping to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/105950 [13:46:15] (03PS2) 10Mark Bergsma: Move smokeping to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/105950 [13:47:36] (03CR) 10Mark Bergsma: [C: 032] Move smokeping to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/105950 (owner: 10Mark Bergsma) [13:51:48] (03PS1) 10Mark Bergsma: Install smokeping (including web frontend) on netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/105951 [13:53:07] (03CR) 10Mark Bergsma: [C: 032] Install smokeping (including web frontend) on netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/105951 (owner: 10Mark Bergsma) [14:35:31] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [14:37:31] (03PS1) 10Mark Bergsma: Fix web interface configuration, add pathnames file [operations/puppet] - 10https://gerrit.wikimedia.org/r/105952 [14:38:38] (03CR) 10Mark Bergsma: [C: 032] Fix web interface configuration, add pathnames file [operations/puppet] - 10https://gerrit.wikimedia.org/r/105952 (owner: 10Mark Bergsma) [14:41:01] (03PS1) 10Mark Bergsma: Fix regex error, ship smokeping.fcgi file [operations/puppet] - 10https://gerrit.wikimedia.org/r/105953 [14:42:11] (03CR) 10Mark Bergsma: [C: 032] Fix regex error, ship smokeping.fcgi file [operations/puppet] - 10https://gerrit.wikimedia.org/r/105953 (owner: 10Mark Bergsma) [14:53:42] (03PS1) 10Hashar: zuul.pp retab + some puppet-lint fix [operations/puppet] - 10https://gerrit.wikimedia.org/r/105958 [14:53:43] (03PS1) 10Hashar: zuul: define push_change_refs has false [operations/puppet] - 10https://gerrit.wikimedia.org/r/105959 [15:07:02] RECOVERY - zuul_service_running on gallium is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/local/bin/zuul-server [15:08:53] (03PS2) 10Dzahn: do not use generated .htaccess from Bugzilla [operations/puppet] - 10https://gerrit.wikimedia.org/r/105944 [15:09:02] (03CR) 10jenkins-bot: [V: 04-1] do not use generated .htaccess from Bugzilla [operations/puppet] - 10https://gerrit.wikimedia.org/r/105944 (owner: 10Dzahn) [15:11:12] hashar: i'm gettting 404s from integration [15:11:21] following the links from jenkins -1 there [15:11:22] mutante: page ? [15:11:25] err URL I mean [15:11:27] https://integration.wikimedia.org/ci/job/operations-puppet-erblint-HEAD/None/console [15:11:32] ah yeah None [15:11:35] https://gerrit.wikimedia.org/r/#/c/105944/2 [15:11:38] LOST [15:11:39] that is because the build disappeared somehow :/ [15:11:55] you have to retrigger the job (new patchset) [15:12:01] PROBLEM - zuul_service_running on gallium is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/bin/python /usr/local/bin/zuul-server [15:12:08] oh [15:12:09] (03PS3) 10Dzahn: do not use generated .htaccess from Bugzilla [operations/puppet] - 10https://gerrit.wikimedia.org/r/105944 [15:12:21] puts a . in the committ message via gerrit web [15:12:35] oop [15:14:00] hashar: ssh to gallium really slow :p [15:14:28] I am not sure why but the new Zuul version now has two processes :/ [15:14:44] 13448 ? Sl 0:00 /usr/bin/python /usr/local/bin/zuul-server [15:14:44] 13449 ? Sl 0:02 \_ /usr/bin/python /usr/local/bin/zuul-server [15:14:48] (03PS2) 10Tim Landscheidt: Tools: Install ksh [operations/puppet] - 10https://gerrit.wikimedia.org/r/66264 (owner: 10Petrb) [15:14:57] ssh: connect to host gallium port 22: Connection timed out [15:15:06] mutante: ahh you need to proxy via a bastion [15:15:17] direct ssh access from outside is no more allowed [15:15:21] i'm on iron [15:16:02] hashar: yea, confirmed, works from other [15:16:06] e.g. fenari [15:16:14] iron ( 208.80.154.151 ) is not in the ferm rules apparently [15:17:05] sooo, want me to stop/start zuul? [15:17:08] a single one [15:17:12] na it works [15:17:16] ok [15:17:17] zuul/cmd/server.py: child_pid = os.fork() [15:17:20] that is from the code [15:17:23] so I guess it is expected [15:17:47] git-daemon should probably have a real user [15:17:50] "115" [15:18:04] hmm [15:18:16] gitdaemon:x:115:65534::/nonexistent:/bin/false [15:18:33] hmm, ok, in ps it doesnt use the name [15:18:37] ahh [15:18:38] as opposed to jenkins [15:20:37] (03PS1) 10Hashar: nrpe: adjust Zuul process monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/105962 [15:20:38] mutante: maybe there is a typo somewhere , i.e. git-daemon versus gitdaemon [15:20:50] and Zuul process check should be fixed by change above [15:21:03] so when jenkins says LOST it is not a FAIL [15:21:06] nods [15:21:15] yeah LOST means that Zuul could not find the build in Jenkins [15:21:17] it worked on ps3, just adding that . [15:21:22] it has no idea what the status is :( [15:21:24] k, yea [15:21:28] modules/contint/templates/default.git-daemon.erb:GIT_DAEMON_USER=gitdaemon [15:21:28] :D [15:22:00] :) [15:25:20] hashar: redirect https://integration.wikimedia.org/ci/job/operations-puppet-erblint-HEAD/None/console to a descriptive wiki page ?:) [15:25:39] hmm, nvm, the repo is in the URL too [15:25:47] but it's always None/console [15:25:55] we should give a better error message indeed [15:26:01] just for the "None" [15:26:42] wonders how easy that is in " Winstone Servlet Engine" [15:29:38] Why is it called Winstone ? "Winstone is the name of a rather large Jamaican man a friend of mine met one night, while he was out clubbing in the Roppongi area of Tokyo. He (my friend) was a little liquored up at the time.... " haha, nice [15:30:08] http://winstone.sourceforge.net/#whatIs [15:45:12] (03PS3) 10coren: Tools: Install ksh [operations/puppet] - 10https://gerrit.wikimedia.org/r/66264 (owner: 10Petrb) [15:46:45] (03CR) 10coren: [C: 032] "Package install" [operations/puppet] - 10https://gerrit.wikimedia.org/r/66264 (owner: 10Petrb) [15:46:58] <^d> ksh? [15:46:59] (03PS4) 10coren: Tool Labs: package libfcgi-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/84288 (owner: 10DrTrigon) [15:47:02] * ^d starts having flashbacks [15:47:10] * ^d shudders uncontrollably [15:47:39] Some people want it. Meh. [15:48:31] (03PS5) 10coren: Tool Labs: package libfcgi-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/84288 (owner: 10DrTrigon) [15:49:31] (03CR) 10coren: [C: 032] "Simple package install" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84288 (owner: 10DrTrigon) [15:49:44] (03CR) 10coren: "Simple package install" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84288 (owner: 10DrTrigon) [15:50:08] (03PS2) 10coren: Tools: Add requested package libjpeg-turbo-progs [operations/puppet] - 10https://gerrit.wikimedia.org/r/105453 (owner: 10Tim Landscheidt) [15:51:43] (03CR) 10coren: [C: 032] "Package addition." [operations/puppet] - 10https://gerrit.wikimedia.org/r/105453 (owner: 10Tim Landscheidt) [15:52:04] (03PS2) 10coren: Tools: Install requested package python-pyexiv2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/105192 (owner: 10Tim Landscheidt) [15:52:19] (03CR) 10coren: [C: 032] "Package addition." [operations/puppet] - 10https://gerrit.wikimedia.org/r/105192 (owner: 10Tim Landscheidt) [15:55:20] ^d: If I search on Gerrit for "file:foobar", the error: "secondary index must be enabled for file:foobar" appears. Can this be enabled? [15:55:47] <^d> After we upgrade to 2.8 [15:55:58] <^d> The secondary index was very experimental and brokeny in 2.7-rc2 [15:56:33] ^d: Nice. Is there an ETA? [15:57:05] <^d> "Sometime in the next month-ish" is about as close as I can give. [15:57:32] Good enough :-). [15:58:06] (03PS2) 10coren: Tool Labs: install fonts for vCat tool [operations/puppet] - 10https://gerrit.wikimedia.org/r/100147 (owner: 10Dapete) [16:00:11] (03PS3) 10coren: Tool Labs: install fonts for vCat tool [operations/puppet] - 10https://gerrit.wikimedia.org/r/100147 (owner: 10Dapete) [16:01:13] (03PS4) 10coren: Tool Labs: install fonts for vCat tool [operations/puppet] - 10https://gerrit.wikimedia.org/r/100147 (owner: 10Dapete) [16:03:17] (03CR) 10coren: [C: 032] "Okay after merge and rebase." [operations/puppet] - 10https://gerrit.wikimedia.org/r/100147 (owner: 10Dapete) [16:09:38] !log upgrading php-wikidiff2 on test.wikimedia.org [16:09:55] Logged the message, Master [16:48:08] (03PS1) 10Ottomata: Fixing configuration url in .conf.example file [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/105972 [16:48:09] (03PS1) 10Ottomata: Updating version to 1.0.1-1 with recent changes [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/105973 [16:48:24] (03CR) 10Ottomata: [C: 032 V: 032] Fixing configuration url in .conf.example file [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/105972 (owner: 10Ottomata) [16:48:39] (03CR) 10Ottomata: [C: 032 V: 032] Updating version to 1.0.1-1 with recent changes [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/105973 (owner: 10Ottomata) [17:03:10] !log added varnishkafka 1.0.1-1 to apt; installed on mobile varnishes [17:03:28] Logged the message, Master [17:08:18] (03PS1) 10Jgreen: add user Sherah Smith (ssmith) and include in fr-tech group [operations/puppet] - 10https://gerrit.wikimedia.org/r/105974 [17:11:07] (03CR) 10Jgreen: [C: 032 V: 031] add user Sherah Smith (ssmith) and include in fr-tech group [operations/puppet] - 10https://gerrit.wikimedia.org/r/105974 (owner: 10Jgreen) [17:16:48] (03CR) 10Andrew Bogott: [C: 032] "works!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105687 (owner: 10Yuvipanda) [17:29:57] andrewbogott: thanks! [17:34:36] (03PS1) 10Ottomata: Updating varnishkafka.conf.erb with xid comment [operations/puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/105977 [17:34:49] (03CR) 10Ottomata: [C: 032 V: 032] Updating varnishkafka.conf.erb with xid comment [operations/puppet/varnishkafka] - 10https://gerrit.wikimedia.org/r/105977 (owner: 10Ottomata) [17:52:29] (03CR) 10CSteipp: [C: 031] "Someone should stay on top of it to make sure upstream doesn't change the config, but keeping all the configs in one place is much, much b" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105944 (owner: 10Dzahn) [17:53:16] !log reedy updated /a/common to {{Gerrit|I98121adf9}}: db1006 back to full steam [17:53:33] Logged the message, Master [17:54:36] (03PS1) 10Reedy: All non wikipedias to 1.23wmf9 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105978 [17:56:40] (03PS2) 10Reedy: Disable Timeline and wikiheiro on votewiki and loginwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105721 [17:56:58] (03CR) 10Reedy: [C: 032] Disable Timeline and wikiheiro on votewiki and loginwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105721 (owner: 10Reedy) [17:57:06] (03Merged) 10jenkins-bot: Disable Timeline and wikiheiro on votewiki and loginwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105721 (owner: 10Reedy) [17:57:48] (03PS2) 10Reedy: Remove DEBUG_LOG usage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105720 [17:57:54] (03CR) 10Reedy: [C: 032] Remove DEBUG_LOG usage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105720 (owner: 10Reedy) [17:58:03] (03Merged) 10jenkins-bot: Remove DEBUG_LOG usage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105720 (owner: 10Reedy) [17:59:53] (03PS2) 10Reedy: Use getRealmSpecificFilename on extension-list-wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105718 [17:59:58] (03CR) 10Reedy: [C: 032] Use getRealmSpecificFilename on extension-list-wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105718 (owner: 10Reedy) [18:00:07] (03Merged) 10jenkins-bot: Use getRealmSpecificFilename on extension-list-wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105718 (owner: 10Reedy) [18:03:55] (03CR) 10Reedy: [C: 04-1] "Needs rebasing" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94447 (owner: 10Ori.livneh) [18:06:29] (03PS2) 10Reedy: Move texvc out of MediaWiki version specific folder [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93118 [18:08:26] !log Deleting old math binaries across mediawiki-installation [18:08:44] Logged the message, Master [18:10:13] !log reedy synchronized wmf-config/ [18:10:31] Logged the message, Master [18:15:28] (03CR) 10Reedy: [C: 032] Move texvc out of MediaWiki version specific folder [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93118 (owner: 10Reedy) [18:15:36] (03Merged) 10jenkins-bot: Move texvc out of MediaWiki version specific folder [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93118 (owner: 10Reedy) [18:16:38] !log Copied math binary to /usr/local/apache/uncommon/bin without version variant [18:16:54] * Reedy kicks morebots [18:16:55] Logged the message, Master [18:17:28] !log reedy synchronized wmf-config/CommonSettings.php 'Switch math to using version independent math binary' [18:17:45] Logged the message, Master [18:43:01] PROBLEM - MySQL Processlist on db1006 is CRITICAL: CRIT 0 unauthenticated, 1 locked, 0 copy to table, 81 statistics [18:45:02] RECOVERY - MySQL Processlist on db1006 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 18 statistics [18:52:58] hey paravoid, do you think you might have time to deploy https://gerrit.wikimedia.org/r/#/c/102817/ with me and nuria today? it's the patch that adds UA to the varnishncsa format string for EventLogging; see commit msg for rationale. [18:54:26] ori: I'm about to go out but I will be back later. maybe ottomata can help? [18:54:52] waaashappening? [18:55:17] ottomata: i was just asking paravoid if he could deploy https://gerrit.wikimedia.org/r/#/c/102817/ w/nuria and me later today; it's the patch to add UA to the varnishncsa format string for EL [18:55:55] I mean, I could do it myself too, but I know randomly mucking around with the varnishes is not hugely appreciated :P [18:55:57] (03CR) 10Faidon Liambotis: [C: 031] Add user-agent header to the format spec of EventLogging's varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 (owner: 10Ori.livneh) [18:56:14] seems trivial to me [18:56:15] ottomata: up for it? [18:56:25] but I'll defer to ottomata as the opsen responsible for analytics :) [18:57:03] love trivial changes, my favorite kind for teh 1st month at work [18:57:47] hi nuria :) [18:58:16] oh yeh [18:58:18] NO PROBS [18:58:18] holaaa [18:58:19] whoa [18:58:20] caps [18:58:26] unintentional caps [18:58:30] yeah, ori, let's do it [18:58:39] frozen keyboard? [18:58:56] heh [18:59:00] ha, slippery pinky [18:59:02] (03PS2) 10Reedy: All non wikipedias to 1.23wmf9 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105978 [18:59:02] LET'S DO IT [18:59:06] lol [18:59:07] (03CR) 10Reedy: [C: 032] All non wikipedias to 1.23wmf9 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105978 (owner: 10Reedy) [18:59:12] * ori fist-pumps [18:59:14] k yeah, hsould be pretty simple, [18:59:15] merging [18:59:17] (03Merged) 10jenkins-bot: All non wikipedias to 1.23wmf9 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105978 (owner: 10Reedy) [18:59:21] nooo [18:59:23] (03PS4) 10Ottomata: Add user-agent header to the format spec of EventLogging's varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 (owner: 10Ori.livneh) [18:59:31] (03CR) 10Ottomata: [C: 032 V: 032] Add user-agent header to the format spec of EventLogging's varnishncsa instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 (owner: 10Ori.livneh) [18:59:36] WHAT NOO? [18:59:39] you said LETS DO IT [18:59:52] later would've been better, but no biggie. let's do it. [18:59:52] don't worry i haven't merged on puppetmaster yet [18:59:55] oh oh [18:59:57] well [18:59:57] haha [18:59:58] sorry [19:00:16] ask (with caps) and you shall receive [19:00:22] grrr! [19:00:27] stupid netsplits [19:00:33] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All non wikipedias to 1.23wmf9 [19:00:37] ori, shall I continue or do you want me to hold off? [19:00:46] greg-g: Deployment done in 30 seconds in the window... [19:00:51] Logged the message, Master [19:01:09] ottomata: 5 mins [19:01:12] k [19:01:15] thanks [19:02:01] greg-g: looking for an LD window today, post-lunch [19:02:23] greg-g: looking to deploy a small patch to abuse filter, and the abuse filter patch to Flow [19:02:43] Reedy: nice [19:02:59] ebernhardson: I've got 1 hour and 58 minutes left in my deploy window... [19:03:00] ebernhardson: cool, do you have the patches yet? [19:03:01] 57 [19:03:04] h3h [19:03:06] heh [19:03:13] #nottryingtobel33tjustatypo [19:03:58] greg-g: yes, https://gerrit.wikimedia.org/r/#/c/104496/ and https://gerrit.wikimedia.org/r/#/c/103109/ [19:04:36] greg-g: but if we deploy post-lunch, there is also a spam blacklist patch for flow we are working on that should be able to go in as well [19:05:07] ebernhardson: cool, wanna go at 2 pacific? [19:05:23] greg-g: sounds good, thanks [19:12:37] (03PS1) 10Hashar: zuul.eqiad.wmnet. IN CNAME gallium.wikimedia.org. [operations/dns] - 10https://gerrit.wikimedia.org/r/105983 [19:13:09] moaar dns :D [19:18:47] (03PS1) 10Ori.livneh: EventLogging: send esams data directly to vanadium using private iface [operations/puppet] - 10https://gerrit.wikimedia.org/r/105987 [19:19:33] (03CR) 10Ori.livneh: "I verified that sending UDP from an esams bits varnish directly to vanadium works." [operations/puppet] - 10https://gerrit.wikimedia.org/r/105987 (owner: 10Ori.livneh) [19:20:40] (03CR) 10Ottomata: [C: 032 V: 032] EventLogging: send esams data directly to vanadium using private iface [operations/puppet] - 10https://gerrit.wikimedia.org/r/105987 (owner: 10Ori.livneh) [19:20:55] thanks [19:27:21] wasnt ssmollet a short term ops labs contractor? [19:27:30] wrong channel [19:27:32] ffs [19:28:02] ori, lemme know when you are ready for a merge [19:30:32] Contract Extended https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Report,_June_2012 Contracts Ended https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Report,_September_2012 [19:31:02] Reedy: lrn2channelswitch [19:31:28] phone fail [19:45:25] !log csteipp synchronized php-1.23wmf9/img_auth.php 'bug 57016' [19:45:42] Logged the message, Master [19:46:34] !log csteipp synchronized php-1.23wmf9/includes 'bug 58699' [19:46:51] Logged the message, Master [19:47:16] !log merged changes to eventlogging varnishncsa instances. user agent added, esams instances now send directly to vanadium instead of gadolinium relay [19:47:34] Logged the message, Master [19:52:02] !log csteipp synchronized php-1.23wmf8/includes 'bug 58699' [19:52:21] Logged the message, Master [19:56:57] (03PS1) 10Reedy: Revert "All non wikipedias to 1.23wmf9" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105992 [19:57:10] (03CR) 10Reedy: [C: 032] Revert "All non wikipedias to 1.23wmf9" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105992 (owner: 10Reedy) [19:57:19] (03Merged) 10jenkins-bot: Revert "All non wikipedias to 1.23wmf9" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105992 (owner: 10Reedy) [19:58:19] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert All non wikipedias to 1.23wmf9 [19:58:38] Logged the message, Master [20:04:48] (03PS1) 10Reza: Enable GuidedTour on translated languages [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/105997 [20:06:22] Reedy: For completeness we have reports that wikimediafoundation.org has no styling [20:06:31] I know [20:06:34] And I just fixed it [20:06:36] By purging [20:06:38] Ah, K [20:06:42] cndiv: ^^ try again? [20:06:43] Reedy: thank you sir! [20:06:46] will do [20:06:48] Ah, cool [20:06:59] back to normal [20:07:32] well actually it still seems like something is off but I can't put my finger on it. I could also be crazy. I don't look at that page very often. [20:07:39] * aude wonders what's happening [20:08:05] like under 'job openings' the we need / help are on different lines for me. [20:08:08] https://bugzilla.wikimedia.org/show_bug.cgi?id=59780 ? [20:08:21] anywho I just wanted to point it out. [20:09:52] aude: Yeah.. A mixture of a few exceptions happening too commmonly [20:10:03] See the mountain range https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&x=0.5&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception>ype=stack&glegend=show&aggregate=1&embed=1 [20:10:13] huh [20:13:12] aude: 3-5 exceptions that were happening fairly frequently [20:13:20] (03PS1) 10Ottomata: Setting up replication of analytics/camus to github wikimedia/camus [operations/puppet] - 10https://gerrit.wikimedia.org/r/106002 [20:14:25] (03CR) 10Ottomata: [C: 032 V: 032] Setting up replication of analytics/camus to github wikimedia/camus [operations/puppet] - 10https://gerrit.wikimedia.org/r/106002 (owner: 10Ottomata) [20:15:35] (03PS1) 10Ottomata: Reverting addition of camus github mirror [operations/puppet] - 10https://gerrit.wikimedia.org/r/106003 [20:16:26] (03CR) 10Ottomata: [C: 032 V: 032] Reverting addition of camus github mirror [operations/puppet] - 10https://gerrit.wikimedia.org/r/106003 (owner: 10Ottomata) [20:25:00] Nemo_bis: Soooo [20:25:33] $t = Title::makeTitleSafe( 102, 'Barry_Answering_Questions_6_-_Computer.theora.ogvx' ); [20:25:33] var_dump( $t ); [20:25:39] The wikis that are failing don't have NS 102 [20:26:07] Which is TimedText [20:26:08] Which is TMH [20:26:58] huh http://en.wikisource.beta.wmflabs.org/wiki/Special:UncategorizedFiles works [20:27:21] why would they not have that [20:27:57] because few wikis need it [20:28:11] !log upgraded wikitech to 1.23wmf9 [20:28:15] (03CR) 10Odder: [C: 04-1] "This patch set should be abandoned." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103107 (owner: 10Yatinmaan) [20:28:22] I suspect Reedy is saying it's my fault because I requested it to be disabled originally :P [20:28:42] Nemo_bis: Not exactly [20:28:52] currently I believe Commons, en.wiki adn pt.wiki have it, or something; maybe the extension assumed it would saty everywhere [20:29:02] Why's it fine on wmf8 though? [20:29:09] We enabled it on pt.wp recently. [20:29:33] some of the recent changes to Title stuff? [20:33:26] So I guess morebots has adopted a strict twice-per-day failure regimen [20:33:34] Someday I will devote the day needed to fixing! [20:34:01] doesn't Tim's restart loop exist any longer at all? [20:34:29] Morebots lives on toollabs now [20:34:32] So, I guess not. [20:35:09] well it was a world-readable shell script, allegedly [20:35:25] we should not have a restart loop [20:35:27] * aude suspects https://git.wikimedia.org/commitdiff/mediawiki%2Fcore.git/65a4ae9fe949be90647af719962ec6092561e77a [20:35:43] no way i can reproduced the exception [20:37:17] !log upgraded wikitech to 1.23wmf9 [20:37:18] !log reedy synchronized php-1.23wmf8/includes/filerepo/file/File.php [20:37:33] Nemo_bis: weirdly, morebots doesn't crash nearly as much in the other channels. Despite running with the same code on the same host. [20:37:59] Wait, who the heck is logmsgbot? [20:38:09] andrewbogott: it's understandable, it has to receive and parse more traffic hasn't it [20:38:20] not a lot more [20:38:35] logmsgbot is deployment stuff from tin [20:38:36] paravoid: ping [20:38:54] andrewbogott: tcpircbot module in puppet [20:39:02] i wrote it so it doesn't crash :P [20:39:04] I wonder if morebots can't write to wikitech... [20:39:24] andrewbogott: it may time out; it needs to resubmit the entire SAL on each log message [20:40:05] morebots, what's your deal? [20:40:45] :( [20:41:04] maybe as a temp. workaround have it acknowledge the !log immediately and spawn a subprocess to actually handle submission to wikitech [20:41:48] this would mean that morebots could acknowledge a !log and then fail to commit it but given its current behavior i don't think anyone is expecting ACID from it [20:43:33] jeremyb: you around ? [20:43:37] !log reedy synchronized php-1.23wmf9/includes/filerepo/file/File.php [20:43:54] (03PS1) 10RobH: replace wildcard cert with standard cert for racktables [operations/puppet] - 10https://gerrit.wikimedia.org/r/106060 [20:44:01] (03CR) 10jenkins-bot: [V: 04-1] replace wildcard cert with standard cert for racktables [operations/puppet] - 10https://gerrit.wikimedia.org/r/106060 (owner: 10RobH) [20:44:37] (03PS2) 10RobH: replace wildcard cert with standard cert for racktables [operations/puppet] - 10https://gerrit.wikimedia.org/r/106060 [20:45:17] RobH: mind deploying a DNS entry for me please https://gerrit.wikimedia.org/r/#/c/105983/ that introduce zuul.eqiad.wmnet. to replace internal uses of integration.wikimedia.org [20:45:36] ori: I got a fatal trying to archive SAL [20:45:41] ori: I can think of many possible approaches, none of which motivate me to work on this today :) [20:45:52] Nemo_bis: that's telling [20:45:55] (03CR) 10Lcarr: [C: 032] nrpe: adjust Zuul process monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/105962 (owner: 10Hashar) [20:46:13] 
 doesn't help
[20:46:33] 	 andrewbogott: it's a fun project with few dependencies, i think that if we documented the specs well, it might attract a volunteer
[20:46:45] 	 yeah, good point.
[20:46:58] 	 And it runs on toollabs now, so easy to give access.
[20:47:02] * ori  nods
[20:47:19] 	 (03CR) 10RobH: [C: 032] replace wildcard cert with standard cert for racktables [operations/puppet] - 10https://gerrit.wikimedia.org/r/106060 (owner: 10RobH)
[20:47:40] 	 well I get a blank page on https://wikitech.wikimedia.org/w/index.php?title=Special%3ASearch&search=Template%3ATlf too
[20:47:44] 	 hashar: can in a momenet, sorry
[20:47:51] 	 hashar: i see unmerged zuul change
[20:47:53] 	 i assume that was you?
[20:47:59] 	 +command[check_zuul]=/usr/lib/nagios/plugins/check_procs -w 2:2 -c 2:2 --ereg-argument-array '^/usr/bin/python /usr/local/bin/zuul-server' 
[20:48:11] * RobH  is merging on palladium
[20:48:20] 	 cuz its live now
[20:48:26] 	 LeslieCarr: yeah
[20:48:36] 	 LeslieCarr: have a bunch more redirects in the pipe btw
[20:48:39] 	 ok
[20:48:46] 	 you have tested ?
[20:48:48] 	 LeslieCarr: is that why you poked? :)
[20:48:53] 	 yes
[20:48:54] 	 :)
[20:48:57] 	 LeslieCarr: do you know what is the MTU between prod hosts?
[20:48:57] 	 i only really tested the php change
[20:49:00] 	 (03CR) 10RobH: "looks good, pushing live per hashar request" [operations/dns] - 10https://gerrit.wikimedia.org/r/105983 (owner: 10Hashar)
[20:49:08] 	 (03CR) 10RobH: [C: 032] "looks good, pushing live per hashar request" [operations/dns] - 10https://gerrit.wikimedia.org/r/105983 (owner: 10Hashar)
[20:49:38] 	 hashar: your dns change is now live
[20:49:45] 	 just updated nameservers
[20:49:47] 	 i.e. how many bytes can i send via UDP and avoid fragmentation
[20:49:59] 	 RobH: the nrpe zuul change was merged by leslie just before you :-D
[20:50:03] 	 LeslieCarr: i first changed the conf manually for 2 entries and tested those. then made a php change to generate the conf as i had manually tweaked. then i regenerated with php change and the generated version for those 2 lines was the same as my tested version 
[20:50:23] 	 wikitech search is borked
[20:50:24] 	 jgage hi!
[20:50:24] 	 RobH: thanks! I will migrate the jenkins jobs so they no more rely on integration.wikimedia.org . That would let us move it to varnish hopefully
[20:50:26] 	 you around?
[20:50:39] 	 hashar: awesome, let me know and then we'll merge your other chagne and get it live!
[20:50:47] 	 well, once i replace the wildcard as well
[20:50:48] 	 ottomata: hi! wanna hangout?
[20:50:55] 	 RobH: well we do that tomorrow :-]  it is to late to do such a change
[20:51:14] 	 manybubbles: ^d - wikitech search is broken
[20:51:16] 	 indeed
[20:51:18] <^d>	 Broken how?
[20:51:18] 	 RobH: you can change the wildcard since internally we use http
[20:51:25] 	 it does not work
[20:51:36] 	 returns blank page
[20:51:38] 	 LeslieCarr: my guess is that its never run its index update stuff and released a bunch of ties
[20:51:40] 	 times
[20:51:47] 	 I wonder if we can get a log for it
[20:51:59] 	 I might've broken wikitech… I just upgraded the mediawiki version
[20:52:03] 	 and did 'submodule update'
[20:52:10] <^d>	 That'd probably do it.
[20:52:24] 	 puppet is updating on magnesium, apache is flapping from this
[20:52:25] 	 ok… which of that should I not have done?
[20:52:33] 	 fyi: RT and racktables access may be spotty for next few minutes
[20:52:48] 	 hey greg-g, has the regular deploy to commons happened yet or has it been delayed?
[20:52:56] <^d>	 andrewbogott: Updated the submodules en masse :)
[20:52:57] 	 andrewbogott: ^d and I manage all updates to cirrussearch because it is still under active development and we keep modifying the Elasticsearch schema
[20:53:00] <^d>	 But we can work around it.
[20:53:15] 	 we're careful about making the schema changes backwards compatible for the versions on the cluster
[20:53:17] 	 hi jgage, yeahhh!
[20:53:18] 	 lets
[20:53:30] 	 but we forget wikitech
[20:53:36] 	 cool, i just reserved a room and send you an invite, be there in a sec
[20:53:37] 	 jgage: https://plus.google.com/hangouts/_/calendar/d2lraW1lZGlhLm9yZ19jYjM3bXU0OGNuaHRkN2hybmE4czI3b25hb0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t.c6j7qidqs491nhi7ovk9pi4h14?authuser=1
[20:53:38] 	 can fix it by rebuilding the search index though
[20:53:41] 	 oh ok cool
[20:53:52] 	 manybubbles, ^d:  OK… wikitech doesn't use the deployment system at the moment so everything is strictly by hand.
[20:53:55] <^d>	 We can actually run the scripts from our end.
[20:54:00] 	 Next time I upgrade… how should I do it?
[20:54:02] <^d>	 with --basePath or w/e it's called
[20:54:03] 	 ^d: we can!
[20:54:22] 	 if you can figure it out then please do and let me know
[20:54:27] 	 then we don't have to bother them
[20:54:42] 	 andrewbogott: or you could read the readme about updating the search config.  there is a big section on it.
[20:55:54] <^d>	 manybubbles: $ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=mediawikiwiki --baseName=wikitechwiki
[20:56:11] 	 will that let us reindex it?
[20:56:36] <^d>	 Probably?
[20:57:01] 	 RECOVERY - zuul_service_running on gallium is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/local/bin/zuul-server  
[20:57:27] 	 LeslieCarr: and zuul nrpe check has been fixed. Thanks!
[20:57:40] 	 yw
[20:57:46] 	 Sorry, ^d and manybubbles, I can't tell which of the above were directed to me.
[20:57:52] 	 jeremyb: all right - here we go apache config files :)  
[20:57:58] 	 (03CR) 10Hashar: "" [operations/puppet] - 10https://gerrit.wikimedia.org/r/105962 (owner: 10Hashar)
[20:58:02] 	 ^d: you wanna give it a shot
[20:58:04] 	 I take it when a 'normal' deployment happens we don't update all modules?
[20:58:10] 	 andrewbogott: I think I'm going to bully ^d into doing it
[20:58:13] <^d>	 manybubbles: I tried it with --reindexAndRemoveOk.
[20:58:14] 	 If not, how is it determined which update and which don't?
[20:58:31] 	 (03CR) 10Lcarr: [C: 032] fix redirects for percent-encoded destinations [operations/apache-config] - 10https://gerrit.wikimedia.org/r/104984 (owner: 10Jeremyb)
[20:58:35] 	 andrewbogott: it is complicated :(
[20:58:38] 	 (03CR) 10Lcarr: [V: 032] fix redirects for percent-encoded destinations [operations/apache-config] - 10https://gerrit.wikimedia.org/r/104984 (owner: 10Jeremyb)
[20:58:44] 	 normally when I deploy I just update what I need
[20:59:03] 	 but weekly we cut a new deployment that pushes all the new code in phases to the wikis
[20:59:05] 	 (03CR) 10Lcarr: [C: 032 V: 032] write a \n at EOF after generating rest of conf [operations/apache-config] - 10https://gerrit.wikimedia.org/r/104985 (owner: 10Jeremyb)
[20:59:10] 	 manybubbles: that's what we've been doing on wikitech for a year, which is why every single extension was out of sync :)
[20:59:11] 	 and those I just have to know are coming
[20:59:23] 	 So search is the one exception?
[20:59:47] <^d>	 For the time being.
[20:59:53] <^d>	 Hopefully we'll stop changing the schema soon.
[20:59:55] 	 hrmmmm, why can't i push?
[21:00:07] 	 ok...
[21:00:15] * andrewbogott  takes no lesson away from this
[21:00:26] <^d>	 The other option is going back to mysql search
[21:01:20] 	 greg-g: I can haz deploy?
[21:01:23] <^d>	 manybubbles: Is it not wikitechwiki? I'm kind of shooting in the dark here :\
[21:01:24] 	 andrewbogott: I'm not sure what to do
[21:01:29] 	 I'm not complaining, was just worried that there was some other upgrade path I should've followed… sounds like the answer to that is 'not really'
[21:01:39] 	 ^d: you can check the list of indexes
[21:02:09] 	 andrewbogott: I think the answer is that wikitech is currently not well supported and we should figure ou thow to fix that or decide that we don't mind breaking it every once in a while
[21:02:27] 	 "we don't mind breaking it every once in a while" for the time being.
[21:02:38] 	 I think that is what we're doing by default now
[21:02:39] 	 bd808: word
[21:02:50] <^d>	 Yep.
[21:02:53] * greg-g  just got back from lunch
[21:02:56] <^d>	 wikitechwiki's the name.
[21:03:06] 	 greg-g: Thanks. Starting now...
[21:03:17] <^d>	 Schema's up to date, don't know why we're getting white pages.
[21:03:21] <^d>	 Logs would be nice.
[21:03:29] 	 !log updating apache redirects via sync-apache 
[21:03:39] 	 (03PS1) 10Jeremyb: clean up chapters redirects [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106107 
[21:03:39] 	 (03PS1) 10Jeremyb: move {jobs,careers}.wikimedia.org to redirects.dat [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106108 
[21:03:40] 	 (03PS1) 10Jeremyb: consolidate wikimedia.org; rm some redirect hops [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106109 
[21:03:41] 	 (03PS1) 10Jeremyb: rm trailing slash from destinations where unneeded [operations/apache-config] - 10https://gerrit.wikimedia.org/r/106110 
[21:05:07] 	 LeslieCarr: i think jenkins won't check those unless you make a change and push. (you're trusted, i'm not)
[21:05:50] 	 LeslieCarr: the chapters one is the only one that's fixing stuff that's now broken. the rest are just cleanup
[21:06:31] 	 greg-g: Done.
[21:07:17] <^d>	 Can someone please check the apache error log to see why php is dying on wikitechwiki?
[21:07:37] 	 Do the log messages from git deploy go anywhere?
[21:07:52] <^d>	 wikitechwiki logs don't go to fluorine :(
[21:08:11] 	 bd808: jesus
[21:08:42] 	 ori: That makes perfect sense
[21:09:47] 	 !log running apache gracefull all 
[21:10:02] 	 ori: https://gerrit.wikimedia.org/r/#/c/104778 https://gerrit.wikimedia.org/r/#/c/104779/ tiny commits
[21:10:13] 	 (03PS1) 10RobH: Replace * cert with rt.wikimedia.org cert [operations/puppet] - 10https://gerrit.wikimedia.org/r/106113 
[21:10:35] 	 So I see nothing in RT setup that installs the star cert, so I had to add my own call
[21:10:38] <^d>	 LeslieCarr: Can you look at the apache error log on wikitechwiki and tell me why search is dying? I can't do much with a blank page and I'm shooting blind without logs.
[21:10:41] 	 kind of annoying.
[21:10:55] 	 ok
[21:11:04] 	 LeslieCarr: woot, kr's working at least
[21:11:29] 	 ^d http://pastebin.com/aSwT2q8u
[21:12:05] 	 is that helpful ? 
[21:12:15] <^d>	 Very. Thanks!
[21:12:51] 	 PROBLEM - DPKG on mw1193 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:13:01] 	 PROBLEM - DPKG on mw103 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:13:15] 	 So im going to roll a change for RT, sorry if I break it in advance ;]
[21:13:24] 	 [Tue Jan 07 21:05:50 2014] [error] [client 10.4.0.115] PHP Fatal error:  Cannot access protected property WikiPage::$mPreparedEdit in /srv/org/wikimedia/controller/wikis/slot0/extensions/SemanticMediaWiki/includes/SMW_ParseData.php on line 477
[21:13:28] 	 that is me, ignore the dpkg
[21:13:28] <^d>	 andrewbogott: What branch of MW did you deploy?
[21:13:28] 	 (03CR) 10Jeremyb: "This fixed bug 54883." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/104984 (owner: 10Jeremyb)
[21:13:51] 	 RECOVERY - DPKG on mw1193 is OK: All packages OK  
[21:13:51] 	 PROBLEM - DPKG on mw1064 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:13:52] 	 PROBLEM - DPKG on mw1093 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:13:52] 	 PROBLEM - DPKG on mw100 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:13:52] 	 PROBLEM - DPKG on mw104 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:13:52] 	 PROBLEM - DPKG on mw121 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:01] 	 ^d: Using branches/tags of SMW won't likely recieve bugfixes for core breakages...
[21:14:01] 	 PROBLEM - DPKG on mw106 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:01] 	 PROBLEM - DPKG on mw102 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:01] 	 RECOVERY - DPKG on mw103 is OK: All packages OK  
[21:14:01] 	 PROBLEM - DPKG on mw107 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:01] 	 PROBLEM - DPKG on mw117 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:03] 	 !log upgraded wikitech to 1.23wmf9
[21:14:09] 	 ^d:   ^
[21:14:16] 	 Alas, morebots is in a snit
[21:14:16] 	 (03PS2) 10RobH: Replace * cert with rt.wikimedia.org cert [operations/puppet] - 10https://gerrit.wikimedia.org/r/106113 
[21:14:21] <^d>	 andrewbogott: ty
[21:14:31] 	 PROBLEM - DPKG on mw1199 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:31] 	 PROBLEM - DPKG on mw1144 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:41] 	 PROBLEM - DPKG on mw1215 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:41] 	 PROBLEM - DPKG on mw1161 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:41] 	 PROBLEM - DPKG on mw1207 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:41] 	 PROBLEM - DPKG on mw1099 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:51] 	 PROBLEM - DPKG on mw1211 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:51] 	 PROBLEM - DPKG on mw1202 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:51] 	 PROBLEM - DPKG on mw1165 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:51] 	 PROBLEM - DPKG on mw1168 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:51] 	 RECOVERY - DPKG on mw1064 is OK: All packages OK  
[21:14:52] 	 RECOVERY - DPKG on mw1093 is OK: All packages OK  
[21:14:52] 	 PROBLEM - DPKG on mw1156 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:53] 	 PROBLEM - DPKG on mw116 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:53] 	 RECOVERY - DPKG on mw100 is OK: All packages OK  
[21:14:54] 	 PROBLEM - DPKG on mw108 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:54] 	 PROBLEM - DPKG on mw120 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:14:55] 	 RECOVERY - DPKG on mw104 is OK: All packages OK  
[21:14:55] 	 RECOVERY - DPKG on mw121 is OK: All packages OK  
[21:15:01] 	 PROBLEM - DPKG on mw1203 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:01] 	 PROBLEM - DPKG on mw1081 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:01] 	 RECOVERY - DPKG on mw106 is OK: All packages OK  
[21:15:01] 	 PROBLEM - DPKG on mw112 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:02] 	 RECOVERY - DPKG on mw102 is OK: All packages OK  
[21:15:02] 	 PROBLEM - DPKG on mw49 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:02] 	 RECOVERY - DPKG on mw107 is OK: All packages OK  
[21:15:03] 	 PROBLEM - DPKG on mw89 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:03] 	 RECOVERY - DPKG on mw117 is OK: All packages OK  
[21:15:04] 	 PROBLEM - DPKG on mw99 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:04] 	 PROBLEM - DPKG on mw1049 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:05] 	 PROBLEM - DPKG on mw5 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:05] 	 PROBLEM - DPKG on mw97 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:06] 	 PROBLEM - DPKG on mw1119 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:11] 	 PROBLEM - DPKG on mw113 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:11] 	 PROBLEM - DPKG on mw19 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:11] 	 PROBLEM - DPKG on mw1183 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:20] 	 ^d manybubbles -- here's  a longer php list of errors -- http://pastebin.com/qZkCpGjK
[21:15:21] 	 PROBLEM - DPKG on mw82 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:21] 	 PROBLEM - DPKG on mw1103 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:31] 	 PROBLEM - DPKG on mw15 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:31] 	 PROBLEM - DPKG on mw9 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:31] 	 PROBLEM - DPKG on mw78 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:31] 	 PROBLEM - DPKG on mw84 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:31] 	 RECOVERY - DPKG on mw1199 is OK: All packages OK  
[21:15:32] 	 PROBLEM - DPKG on mw1182 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[21:15:32] 	 RECOVERY - DPKG on mw1144 is OK: All packages OK  
[21:15:44] <^d>	 It's the same thing.
[21:16:15] <^d>	  /srv/org/wikimedia/controller/wikis/slot0/extensions/Elastica/Elastica/lib/Elastica/Query/FunctionScore.php - it's claiming the file doesn't exist.
[21:16:18] 	 (03CR) 10RobH: [C: 032] "man i hope this works ;]" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106113 (owner: 10RobH)
[21:16:23] <^d>	 Definitely should as of wmf9
[21:16:32] 	 ^d: that is the one that we had to add the extra import
[21:16:47] 	 hey, I bet the Elastica submodule inside the elastica plugin didn't update right
[21:16:52] 	 because sometimes it doesn't for me
[21:16:58] 	 ^d, how would things get into slot0?
[21:17:21] 	 updating rt, it may flap during this update
[21:17:24] 	 Hm, I guess it looks up to date in any case...
[21:17:31] <^d>	 andrewbogott: Ah, try this. Go to the repos, then `git submodule update --init --recursive`
[21:18:11] 	 done
[21:18:47] <^d>	 Hmm, still nothing.
[21:18:56] 	 Reedy: anyone looking at the wmf9 fatal you provided the stacktraces for https://bugzilla.wikimedia.org/show_bug.cgi?id=59780 ? I'm just catching up after lunch
[21:19:20] 	 Not actively I don't think... I've had a look around and aude did too
[21:19:36] 	 i don't have an answer
[21:19:45] 	 A few other unrelated exceptions that are mostly fixed up too now
[21:19:47] 	 Just that one
[21:19:51] * greg-g  nods
[21:20:09] * aude  has suspicions
[21:20:16] 	 aude: so, I see api in there near the top, is that me being dumb or should anomie look?
[21:20:43] <^d>	 hmm.
[21:20:44] 	 can't create items on test.wikidata
[21:20:54] 	 greg-g: ?
[21:20:59] 	 might be unrelated but has to do with spam blacklist maybe
[21:21:02] 	 (03PS1) 10RobH: Replace * cert with rt.wikimedia.org cert [operations/puppet] - 10https://gerrit.wikimedia.org/r/106114 
[21:21:06] 	 anomie: re https://bugzilla.wikimedia.org/show_bug.cgi?id=59780
[21:21:37] 	 durn whitespaces
[21:21:38] * anomie  looks
[21:21:39] 	 (03PS2) 10RobH: Replace * cert with rt.wikimedia.org cert [operations/puppet] - 10https://gerrit.wikimedia.org/r/106114 
[21:21:48] 	 that was a nice break
[21:22:36] 	 o_0
[21:22:41] 	 hehe
[21:22:59] 	 the bots gone sentient... skynet is now!
[21:23:11] 	 greg-g: At first glance, I blame TimedMediaHandler for calling ApiQueryAllPages without trying to catch the exception. But I'm continuing to look into it.
[21:23:18] * greg-g  nods
[21:23:23] 	 that's the next step yeah :)
[21:23:23] 	 (03CR) 10RobH: [C: 032] Replace * cert with rt.wikimedia.org cert [operations/puppet] - 10https://gerrit.wikimedia.org/r/106114 (owner: 10RobH)
[21:23:35] 	 There's been some changes to TMH
[21:23:45] 	 But not anything obviously related
[21:23:46] * anomie  wonders WhyTF TimedMediaHandler is calling ApiQueryAllPages instead of just doing the necessary query anyway
[21:23:55] 	 code reuse!
[21:23:58] 	 :)
[21:24:09] 	 https://bugzilla.wikimedia.org/show_bug.cgi?id=59797 for creating items
[21:24:16] 	 related to special page
[21:24:23] <^d>	 anomie: Same reason it was using the API to get an image thumbnail.
[21:24:36] 	 ^d: Which is?
[21:25:07] <^d>	 ...
[21:25:13] <^d>	 I'll let you know once I figure it out.
[21:25:34] 	 maybe it was migrated from some fancy Commons JS script :D
[21:25:40] 	 it's tryint to make a wikipage out of a title that is the special page (for wikidata)
[21:25:59] * aude  wonders what/how that changed
[21:29:05] 	 so nothing changed in spam blacklist and test.wikidata uses same version of wikibase
[21:29:21] * aude  can make items on wikidata but not test.wikidata, so it must be a core issue
[21:32:29] 	 greg-g: ^
[21:32:43] * aude  thinks that's kind of a blocker for wikidata to have wmf9
[21:32:45] 	 greg-g: So WhyTF is TimedMediaHandler passing 102 for apnamespace on metawiki? metawiki doesn't even have a namespace 102.
[21:32:53] 	 will see if we can fix
[21:32:54] 	 I'm sorry, RobH. I'm afraid I can't do that
[21:34:12] 	 i think it's an issue in spam blacklist btw
[21:34:46] 	 aude: yeah, agree re blocker
[21:35:03] 	 it's probably wikidata specific
[21:35:35] 	 ^d, still working on wikitech search, or did you surrender?
[21:36:15] 	 hmm, no maxsem
[21:36:26] 	 (03CR) 10Hashar: "Jenkins jobs change is https://gerrit.wikimedia.org/r/106116" [operations/dns] - 10https://gerrit.wikimedia.org/r/105983 (owner: 10Hashar)
[21:36:54] 	 greg-g: Uhh. "ForeignApiQueryAllPages". It's going to try to construct a Title object against the local wiki and then query it against some other wiki's database?
[21:36:59] * anomie  git blames, and finds Jan Gerber is to blame for that mess.
[21:37:36] 	 :(
[21:38:19] 	 !jenkins integration-jjb-config-test
[21:38:19] 	 https://integration.wikimedia.org/ci/job/integration-jjb-config-test
[21:38:47] <^d>	 andrewbogott: I'm not sure what's going on.
[21:39:08] <^d>	 I'm assuming we're still getting the same "file not found" error?
[21:40:41] 	 failed to open stream: No such file or directory in /srv/org/wikimedia/controller/wikis/slot0/includes/AutoLoader.php on line 1220, referer: https://wikitech.wikimedia.org/wiki/Main_Page
[21:40:42] 	 anomie: just let me know if you get to a stuck point
[21:40:45] 	 So, I think that's yes
[21:42:00] 	 greg-g: I updated the bug with details and reassigned it to TimedMediaHandler.
[21:43:37] 	 anomie: thanks
[21:43:51] 	 I do wonder why it isn't broken in wmf8
[21:45:06] 	 Reedy: Gerrit change 40337 had the side effect of having the API call Title::makeTitleSafe with the passed namespace number, instead of Title::newFromText while ignoring the namespace number entirely (which caused bug 25702, which is why the change)
[21:45:53] 	 Ah
[21:45:58] 	 That took a while to be merged
[21:46:41] 	 Yeah, that it did. It had a "-1, wait a few months, fix, repeat" development cycle, unfortunately
[21:47:04] 	 !log reedy updated /a/common to {{Gerrit|I9416c2674}}: Revert "All non wikipedias to 1.23wmf9"
[21:47:10] 	 (03PS1) 10Reedy: votewiki really doesn't need TMH for anything [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106119 
[21:47:12] 	 lies all lies
[21:47:34] <^d>	 andrewbogott: I'm going to work around it in wmf9 and above.
[21:47:43] <^d>	 Give me a bit, then should be able to update.
[21:47:49] 	 ok
[21:48:04] * Reedy  goes to find a drink and a snack
[21:48:15] 	 Will poke TMH when I get back
[21:48:28] 	 Reedy: thanks
[21:48:52] 	 Annoyingly it's enabled everywhere
[21:49:02] 	 :(
[21:49:13] 	 alright, now on to aude's thing (haven't even looked yet)
[21:49:56] 	 There's a Flow fix to be deployed too
[21:50:03] 	 But is scheduled I believe
[21:50:05] 	 yes
[21:50:08] 	 in 10 minutes
[21:50:09] 	 we can probably do a revert
[21:50:41] 	 aude: k
[21:50:54] 	 (03PS1) 10Lcarr: adding secondary IP for aluminium for fundraising.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/106120 
[21:55:29] <^d>	 greg-g: Can I sync the Elastic library? Not changing any code, but just switched from being a submodule to not.
[21:55:37] 	 !log removed unified certificate off misc-web-lb servers cp1034/cp1044, there was no puppet entries for their installs, must have been leftover cruft from varnish role change
[21:55:52] 	 Awesome
[21:55:52] 	 class ForeignApiQueryAllPages extends ApiQueryAllPages {
[21:56:08] 	 (03CR) 10Lcarr: [C: 032] adding secondary IP for aluminium for fundraising.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/106120 (owner: 10Lcarr)
[21:56:12] 	 yeah, saw that
[21:56:23] 	 ^d: promise it'll work? :)
[21:56:34] 	 ^d: not right now, though, ebernhardson is fixing Flow
[21:56:43] <^d>	 I promise it will :)
[21:56:49] <^d>	 Ok, well I'm already staged.
[21:56:52] <^d>	 I just won't sync.
[21:56:53] 	 k
[21:57:13] 	 aude, greg-g: Is this bug 59788? It looks like what's going on there is that Wikibase has been passing a context for a special page to the EditFilterMergedContent hook, and Gerrit change 101490 to SpamBlacklist made the reasonable assumption that the context for that hook would be for a page that's actually editable.
[21:57:34] 	 (03PS1) 10Lcarr: giving secondary ip to aluminum [operations/puppet] - 10https://gerrit.wikimedia.org/r/106125 
[21:58:21] 	 https://gerrit.wikimedia.org/r/#/c/106123/ for nowhttps://gerrit.wikimedia.org/r/#/c/106123/
[21:58:33] 	 $title was always invalid for wikibase but was unused
[21:58:46] 	 gah
[21:58:51] <^d>	 andrewbogott: Ok, so this is kind of annoying, but it should be the only time we have to do this. Updating the wmf9 branch. You should get some complaints from the submodule update about not being able to overwrite untracked files (this is fine). Then go into extensions/Elastica, `rm -R Elastica` then `git reset --hard 4d4d3c7e0a6aca7a0429377dfe11ffd973cc31a1`
[21:59:02] <^d>	 s/Updating/Update/
[22:00:28] 	 would be good to have a nicer, real fix than above  asap
[22:00:46] 	 ^d:  After all that should I do a submodule update again?
[22:00:52] 	 jgage: https://rt.wikimedia.org/Ticket/Display.html?id=6602
[22:00:53] 	 :)
[22:01:12] <^d>	 andrewbogott: You can, but it shouldn't be necessary.
[22:01:16] 	 Ok.
[22:01:18] 	 Seems better!
[22:01:21] <^d>	 That 4d4d3c7 is the sha1 we want :)
[22:01:30] <^d>	 All better :)
[22:01:46] 	 ^d:  Thanks, sorry I broke everything
[22:01:50] <^d>	 manybubbles: You were right from the start, it was the submodule not updating right. Workaround is ugly, but it works.
[22:01:56] 	 (03CR) 10Lcarr: [C: 032] giving secondary ip to aluminum [operations/puppet] - 10https://gerrit.wikimedia.org/r/106125 (owner: 10Lcarr)
[22:02:01] 	 anomie: Part of me just wants to steal the old titlePartToKey and copy it into ForeignApiQueryAllPages to override it
[22:02:06] 	 ebernhardson: go forth and fix flow, btw :)
[22:02:14] <^d>	 andrewbogott: No worries...we've kind of hit this problem before with sub-submodules. I've worked around it indefinitely so it shouldn't happen again.
[22:02:15] 	 ^d:I'm pretty awesome.
[22:02:36] <^d>	 andrewbogott: You wouldn't have known, and shouldn't have expected it, tbh :)
[22:02:44] 	 greg-g: ok, i was holding off but will do :)
[22:02:46] 	 Reedy: What really should be done there, IMO, is to skip the API module at all and just call the necessary database query. That's really all ApiQueryAllPages does in the first place.
[22:04:23] 	 bblack: hey. you about ?
[22:05:11] 	 god I hate you google plus
[22:06:53] 	 (03PS1) 10Lcarr: giving the CORRECT secondary ip to aluminium - rt 6601 [operations/puppet] - 10https://gerrit.wikimedia.org/r/106130 
[22:07:37] 	 (03CR) 10Lcarr: [C: 032 V: 032] giving the CORRECT secondary ip to aluminium - rt 6601 [operations/puppet] - 10https://gerrit.wikimedia.org/r/106130 (owner: 10Lcarr)
[22:07:54] 	 god i hate that lcarr person who mis-pastes ip's ;)
[22:11:12] 	 (03PS1) 10RobH: etherpad to use own cert, not wildcard [operations/puppet] - 10https://gerrit.wikimedia.org/r/106131 
[22:12:19] 	 Something is very wrong with wikitech
[22:12:55] 	 kaldari: outside of borked search?
[22:13:09] 	 Everytime I try to save a page I just get a blank screen
[22:13:13] 	 and it doesn't save
[22:13:32] 	 same
[22:13:41] 	 andrewbogott: you still working on wikitech?
[22:14:12] 	 last i knew i thoguht andrew and chad were poking at things
[22:14:21] 	 but i may be mistaken, i wasn't directly involved.
[22:15:20] 	 (03CR) 10RobH: [C: 032] etherpad to use own cert, not wildcard [operations/puppet] - 10https://gerrit.wikimedia.org/r/106131 (owner: 10RobH)
[22:19:00] 	 Reedy: ping
[22:19:14] 	 Hi
[22:19:36] 	 I am trying to cherry-pick a commit for AbuseFilter
[22:20:22] 	 remote: Branch refs/heads/wmf/1.23wmf9:
[22:20:22] 	 remote: You are not allowed to perform this operation.
[22:20:22] 	 remote: To push into this reference you need 'Push' rights.
[22:21:23] 	 What command are you using? Noting the commits need to still go via gerrit...
[22:21:56] 	 Reedy: git push origin wmf/1.23wmf9
[22:22:01] 	 Right, which is wrong
[22:22:09] 	 You know you can do cherry picks via gerrit, right?
[22:22:11] 	 refs/for/wmf/1.23wmf9
[22:22:40] 	 "Cherry Pick To" is pretty useful
[22:25:22] 	 Reedy, ori: thx
[22:26:28] 	 aude: Do we need to wait on a commit for Wikidata stuff?
[22:28:10] 	 !log reedy synchronized php-1.23wmf9/extensions/TimedMediaHandler/
[22:29:26] 	 !log demon synchronized php-1.23wmf9/extensions/Elastica/
[22:30:13] <^d>	 greg-g: ^ that's the change I said wouldn't do anything. it didn't.
[22:30:16] 	 Reedy: yes
[22:30:30] 	 ^d: sweet
[22:30:32] 	 (03CR) 10Ori.livneh: [C: 04-1] "Bryan, do you think you could split this patch in two, with one patch provisioning Kibana and the other patch configuring it to run behind" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis)
[22:30:34] 	 you guys are still workign on fixing wikitech right?
[22:30:38] 	 =P
[22:30:38] 	 bd808: ^
[22:30:42] <^d>	 RobH: Nope, all fixed.
[22:30:45] 	 yay!
[22:30:58] 	 Reedy: if you want to update spam blaclist to master, that is what we need
[22:31:06] 	 ^d: negatory
[22:31:10] 	 edit and save still does blank page
[22:31:13] 	 blacklist*
[22:31:16] 	 ori: That sounds like a good idea.
[22:31:19] <^d>	 RobH: Not search.
[22:31:21] 	 wikitech =! working
[22:31:24] <^d>	 I think it's SemanticSomething.
[22:31:30] 	 != even
[22:31:37] 	 ori: I'm working on fixing the logstash module right now. :/
[22:31:48] 	 yeah, no rush obviously
[22:32:01] 	 ugh
[22:32:10] 	 well, when was wikitech last updated?
[22:32:19] <^d>	 Earlier today :p
[22:32:23] 	 by whom?
[22:32:23] <^d>	 Hence, it broke.
[22:32:42] <^d>	 andrewbogott did. We fixed search.
[22:32:43] 	 shouldn't wikitech be like, security udpates only?
[22:32:51] 	 Etherpad looks down
[22:32:57] 	 andrewbogott: hey, still working on fixing wikitech?
[22:33:07] 	 marktraceur: could be temporary, robh is changing certs
[22:33:13] 	 Ah, 'kay
[22:33:15] 	 !log i broke etherpad
[22:33:16] 	 (03PS1) 10RobH: etherpad to use own cert, not wildcard [operations/puppet] - 10https://gerrit.wikimedia.org/r/106139 
[22:33:17] 	 fixing now
[22:33:18] 	 see
[22:33:38] 	 (03CR) 10RobH: [C: 032 V: 032] etherpad to use own cert, not wildcard [operations/puppet] - 10https://gerrit.wikimedia.org/r/106139 (owner: 10RobH)
[22:33:41] 	 PROBLEM - HTTP on zirconium is CRITICAL: Connection refused  
[22:34:10] 	 runnign update now, sorry folks!
[22:34:17] 	 i stupidly didnt include the damned cert in my patchset.
[22:34:21] <^d>	 greg-g: There was some fatal earlier about SemanticEditFormsSomethingOrAnother earlier.
[22:34:28] * RobH  has done too many near identical patches today
[22:34:28] <^d>	 Which would explain edits failing.
[22:34:41] 	 ugh
[22:35:02] 	 alright, andrewbogott hey, sorry, since you updated wikitech, you're on point for fixing or reverting, k?
[22:35:04] 	 Life is exciting in this channel
[22:35:23] 	 of course when i break it, it takes forever to do a run to fix it
[22:35:39] <^d>	 greg-g: Actually, I think it was that core updated but SemanticWhatever's still pinned at an old version.
[22:35:42] <^d>	 Would explain it.
[22:35:59] 	 why was wikitech updated again? :)
[22:36:09] 	 it ain't a test wiki :)
[22:36:28] * RobH  is going to have to revert his etherpad changes cuz its not fixing as it should
[22:36:32] 	 bah.
[22:36:48] 	 a revert for you, a revert for you, a revert for you, everyone gets a revert
[22:36:57] 	 wth
[22:37:05] 	 ha
[22:37:07] 	 fixed!
[22:37:13] 	 !log etherpad workin
[22:37:16] 	 marktraceur: confirm?^
[22:37:24] 	 Confirmed
[22:37:31] 	 also, no logmsgbot?
[22:37:38] * greg-g  hangs head
[22:37:41] 	 RECOVERY - HTTP on zirconium is OK: HTTP OK: HTTP/1.1 200 OK - 163463 bytes in 0.012 second response time  
[22:37:46] 	 not that it could save the edit anyways....
[22:37:56] 	 oh yea...
[22:38:01] 	 my admin logs are atleast tweeting
[22:38:04] 	 but damn it
[22:38:05] 	 ^d: mind taking a look more deeply? andrewbogott see awol
[22:38:05] 	 >_<
[22:38:15] 	 seems*
[22:38:23] 	 we can text him =]
[22:38:46] 	 the curse of being ops, sure
[22:38:57] 	 me updating my submodules but might be easier / quicker for Reedy  to update spamblacklist
[22:38:57] 	 i'll drop him an sms asking him to hop online if he can
[22:38:57] 	 unless ^d seems like he can handle it quickly
[22:39:12] <^d>	 We don't need andrew specifically.
[22:39:17] <^d>	 We just need to update the submodules.
[22:39:22] <^d>	 To something that's hopefully fixed.
[22:39:39] 	 well, right, but *who*
[22:40:03] * andrewbogott  stepped away, reading backscroll...
[22:40:03] 	 i have root but no clue what to do ;]
[22:40:10] 	 andrewbogott: wikitech wont take edits 
[22:40:21] 	 semanticosmethingorother
[22:40:31] 	 seems related to your update, or atleast timed in same window
[22:40:40] 	 well, nothing else changed during that time :)
[22:41:00] 	 my version of 'omg its all yer fault' was nicer ;]
[22:41:09] 	 just fyi: when we're done, I'mma gonna ask why we decided to update wikitech outof band and unplanned :)
[22:41:21] <^d>	 Ugh, the pastebins expired.
[22:41:29] 	 Hm, looking...
[22:41:33] <^d>	 greg-g: Well it's always out of band.
[22:41:44] 	 ^d: touche, just a fun phrase I guess
[22:42:22] <^d>	 andrewbogott: Can you check the fatal log? I'm guessing we're still seeing something about SemanticSomething?
[22:42:26] <^d>	 The earlier pastebins expired :\
[22:43:51] 	 (03PS1) 10BryanDavis: Fix file permissions and command args for logstash [operations/puppet] - 10https://gerrit.wikimedia.org/r/106142 
[22:43:58] 	 The message is PHP Fatal error:  Cannot access protected property WikiPage::$mPreparedEdit in /srv/org/wikimedia/controller/wikis/slot0/extensions/SemanticMediaWiki/includes/SMW_ParseData.php on line 477
[22:44:03] 	 so, probably the same as what you saw.
[22:44:20] 	 Made protected in core
[22:44:27] 	 Ryan didn't fork semanticstuff for.... oh
[22:44:30] 	 Still using direct access in branch (and mayb ecore)
[22:44:41] 	 s/ecore/master/
[22:44:43] 	 ok, it sounds like y'all are way ahead of me...
[22:44:46] 	 catch me up?
[22:45:42] 	 So, a change was made to core to fix up usages of direct member variable access
[22:46:00] 	 Which broke smw, I take it?
[22:46:02] 	 Hence "Cannot access protected property WikiPage::$mPreparedEdit"
[22:46:03] 	 Right
[22:46:18] 	 Master of SMW might (or might not) have been updated to deal with this change
[22:46:21] 	 (03PS2) 10Ori.livneh: Fix file permissions and command args for logstash [operations/puppet] - 10https://gerrit.wikimedia.org/r/106142 (owner: 10BryanDavis)
[22:46:26] 	 And smw is essentially unmaintained at this point?
[22:46:43] <^d>	 No, it's maintained.
[22:46:51] 	 But irregardless of that, the used branch probably doesn't have the fix
[22:46:51] 	 'SemanticMediaWiki' => '1.8.x',
[22:46:54] 	 So this is just the curse of wikitech using an extension that isn't on en?
[22:47:21] 	 Reedy: are you deploying SpamBlacklist
[22:47:22] 	 (03CR) 10Ori.livneh: [C: 032] Fix file permissions and command args for logstash [operations/puppet] - 10https://gerrit.wikimedia.org/r/106142 (owner: 10BryanDavis)
[22:47:35] 	 1.8.x
[22:47:35] 	  Last updated 8 months ago by jqnatividad.
[22:47:37] <^d>	 andrewbogott: No, curse of us having a super old branch specified in config + an extension we don't use outside wikitech
[22:47:39] 	 https://github.com/wikimedia/mediawiki-extensions-SemanticMediaWiki/branches
[22:47:42] 	 (03CR) 10Ori.livneh: [V: 032] Fix file permissions and command args for logstash [operations/puppet] - 10https://gerrit.wikimedia.org/r/106142 (owner: 10BryanDavis)
[22:47:48] 	 what chad said
[22:47:55] * greg-g  hates tab completing the ^
[22:48:05] 	 "don't use outside wikiteceh" == "isn't on en" :)
[22:48:39] 	 Need to see if it's fixed in master, and backport
[22:48:43] 	 Which may upset the SMW people
[22:48:51] 	 As it'll break some stupid back compat no doubt
[22:48:52] 	 !log reedy synchronized php-1.23wmf9/extensions/SpamBlacklist
[22:49:00] 	 thanks Reedy 
[22:49:19] 	 Why backport rather than just use a newer version of smw on wikitech?
[22:50:18] 	 Ryan was using the branches to keep introduction of new security bugs etc low
[22:50:19] 	 untested mostly, probably
[22:50:28] 	 1.9.X hasn't been apparently touched in 8 months either
[22:50:36] 	 :/
[22:50:56] 	 not that we don't run old code on enwiki either (*cough* CategoryTree *cough*)
[22:51:10] 	 Cannot access protected property WikiPage::$mPreparedEdit in /srv/org/wikimedia/controller/wikis/slot0/extensions/SemanticMediaWiki/includes/SMW_ParseData.php on line 477
[22:51:12] * Reedy  looks at master
[22:51:13] 	 So, in the meantime should I roll back to wmf8?  Or is it too late for that due to db schema changes?
[22:51:33] 	 AFAIK there's no schema changes between wmf8 and wmf9
[22:51:37] 	 So should be good to rollback
[22:52:00] 	 https://github.com/wikimedia/mediawiki-extensions-SemanticMediaWiki/blob/master/includes/SMW_ParseData.php
[22:52:04] 	 File isn't 477 lines long now :(
[22:52:07] 	 !log bsitu synchronized php-1.23wmf9/extensions/AbuseFilter  'Update AbuseFilter'
[22:52:22] 	 And no mention of mPreparedEdit either
[22:52:23] 	 Does wmf8 predate the core change that's causing this problem?
[22:52:49] 	 !log bsitu synchronized php-1.23wmf9/extensions/Flow  'Update Flow to master'
[22:53:13] 	 andrewbogott: what was the previous version on wikitech?
[22:53:32] 	 should revert to a known good :)
[22:53:33] 	 1.23wmf3
[22:53:53] 	 Well, 1.23wmf3 isn't exactly a known good… that's why I upgraded.
[22:54:00] 	 Oh
[22:54:01] 	 :/
[22:54:04] 	 It was Aaron|home that broke it
[22:54:04] 	 https://github.com/wikimedia/mediawiki-core/commit/a87989717086b114e0acf3353160063de1914114
[22:54:17] 	 2013-11-14
[22:54:17] 	 That is, the latest OpenStackManager extension can't run on 1.23wmf3.
[22:54:19] 	 wmf8 won't work either
[22:54:25] 	 But I can fix that more easily than I can fix core and smw
[22:54:41] 	 Bleugh
[22:54:44] 	 I don't love the idea of being trapped on a dead branch...
[22:54:47] 	 ori merged that commit into 1.23wmf4
[22:55:01] 	 It might be easiest to fix SMW
[22:55:23] 	 Though, Aaron|home made it protected... but no accessor added at the same time
[22:55:26] 	 I wonder if there is one
[22:55:59] 	 Awesome
[22:56:06] 	 FlaggedRevs would seem to be broken also
[22:56:06] 	 	public function preloadPreparedEdit( WikiPage $page ) {
[22:56:06] 	 		$this->mPreparedEdit = $page->mPreparedEdit;
[22:56:06] 	 	}
[22:56:33] 	 Reedy: I had a patch to SMW which was rejected for not being b/c
[22:56:43] 	 xD
[22:56:45] 	 Awesome
[22:56:48] 	 (e.g. older MW versions)
[22:56:49] 	 Sounds about right
[22:56:57] 	 b/c?
[22:57:07] 	 back compat to the stone age
[22:57:09] 	 ah
[22:57:19] 	 Aaron|home: I guess you should fix FR at least ;)
[22:57:25] 	 wth
[22:57:29] <^d>	 Or we could just revert the core change.
[22:57:44] 	 Well, partially
[22:57:52] 	 Make it public again
[22:57:54] 	 maybe, though using that field directly is unreliable
[22:57:57] 	 Leave the other usages alone
[22:58:01] 	 but could be done for a moment
[22:58:35] 	 Is it more unreliable now than it was for the preceeding years?
[22:59:53] 	 greg-g: reviewing everything, it seems while we successfully updated AbuseFilter, the Flow update only got half the patches.  Is it ok for us to do another sync-dir (our window is over)
[23:00:05] 	 yeah
[23:00:08] 	 Partial revert in https://gerrit.wikimedia.org/r/106146
[23:00:09] 	 ok thanks
[23:00:30] 	 Reedy: want me to test that patch on wikitech?
[23:00:46] 	 Cherry pick to wmf9 in https://gerrit.wikimedia.org/r/106147
[23:01:14] <^d>	 You haz merges.
[23:01:16] 	 ^d just merged it... so if you git pull
[23:01:26] 	 ok
[23:02:55] 	 So... if we just wait on Flow now for wmf9 in production...
[23:02:58] 	 Then CategoryTree
[23:03:32] 	 PHP Notice:  DB transaction callbacks still pending (from User::invalidateCache, Title::invalidateCache, User::invalidateCache, User::invalidateCache, Title::invalidateCache, User::invalidateCache, Title::invalidateCache).
[23:03:35] 	 These get better and better
[23:03:42] 	 Editing works on wikitech now...
[23:04:04] 	 Reedy: wait, seeing that where?  Not on wikitech?
[23:04:09] 	 Nope
[23:04:11] 	 On the cluster
[23:04:25] 	 Noise we mostly ignore
[23:04:29] 	 Doesn't stop them being amusing
[23:06:04] 	 (03CR) 10Reedy: [C: 032] votewiki really doesn't need TMH for anything [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106119 (owner: 10Reedy)
[23:06:13] 	 (03Merged) 10jenkins-bot: votewiki really doesn't need TMH for anything [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106119 (owner: 10Reedy)
[23:07:01] 	 !log reedy updated /a/common to {{Gerrit|I2ec865afd}}: votewiki really doesn't need TMH for anything
[23:07:07] 	 Logged the message, Master
[23:07:14] 	 (03PS1) 10Reedy: All non wikipedias to 1.23wmf9 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106149 
[23:07:20] 	 OK, so… given that Aaron|home isn't happy with that partial revert, do I need to log some bugs for smw and FlaggedRevs?
[23:08:03] 	 smw master doesn't use that code.. Or at least, not in the same place
[23:08:46] 	 ok, so I just need to do some testing with smw master
[23:09:08] 	 What about FlaggedRevs?
[23:10:26] 	 Thanks for the quick fix, btw, Reedy.
[23:11:31] 	 greg-g: Now, on to mortaility and morbidity...
[23:11:46] 	 I'm honestly not sure what to do about SMW.
[23:12:02] 	 I suspect if you change SMW, SRF and Validator will need updating to newer versions
[23:12:06] 	 The big picture is:  wikitech has always been a seat-of-the-pants operation which Ryan upgraded irregularly and as needed.  I would /love/ for it to join the deployment train so that I don't have to hand-manage it anymore.
[23:12:12] 	 andrewbogott: that gives errors?
[23:12:18] 	 FR?
[23:12:22] 	 it subclasses WikiPage, so protected should not matter
[23:12:42] 	 That's scary
[23:12:56] 	 Aaron|home: OK, I only mention it because, um  "Reedy: FlaggedRevs would seem to be broken also"
[23:13:01] 	 Mmm
[23:13:19] 	 "Because I'm a subclass of the class you are, I can access your protected members"
[23:13:22] 	 Just seems weird
[23:13:42] 	 Reedy: That's how protected members have always worked?  In c++ at least
[23:14:00] 	 php's visibility features contribute absolutely nothing useful, they're strictly a cause for bugs
[23:14:05] 	 lols
[23:14:14] 	 !log bsitu synchronized php-1.23wmf9/extensions/Flow  'Update Flow to master'
[23:14:15] 	 andrewbogott: Yeah, I think it is for other languages too. Still seems weird
[23:14:16] 	 Reedy: that is the whole point of protected
[23:14:20] 	 Logged the message, Master
[23:14:22] 	 if i had the stamina for it i'd argue that we should use 'var' and 'function' without visibility declarations exclusively
[23:14:23] 	 otherwise you would use private
[23:14:30] 	 no, protected is itself and subclasses
[23:14:37] 	 andrewbogott: it can't join the train, it has to be on separate machines so when the cluster goes down...
[23:14:40] 	 Not through an instance of another subclass
[23:14:51] 	 greg-g: there are middle grounds
[23:14:55] 	 probably
[23:14:57] 	 greg-g: like deployment train w/static mirroring
[23:15:00] * greg-g  nods
[23:15:02] 	 Just because you're a sibling of a parent, doesn't mean you can access sibling properties
[23:15:19] 	 Fuck OOPS
[23:15:21] 	 ori: that's a good one, really
[23:15:28] 	 Reedy: meh, it's useful
[23:15:38] 	 greg-g:  wikitech-static is there for emergencies
[23:15:40] 	 actually FileBackendMultiWrite relies on that a fair bit ;)
[23:15:56] 	 andrewbogott: how often is that sync'd over?
[23:16:01] 	 when someone remembers
[23:16:16] 	 heh, cron?
[23:16:25] 	 I don't know.
[23:16:29] 	 hopefulyy that person's name is cron
[23:16:34] 	 bsitu: All done with flow now?
[23:16:52] 	 PHP Warning:  array_shift() expects parameter 1 to be array, null given in /usr/local/apache/common-local/php-1.23wmf9/extensions/Flow/includes/Repository/TreeRepository.php on line 243
[23:17:02] 	 PHP Warning:  header() expects parameter 3 to be long, string given in /usr/local/apache/common-local/php-1.23wmf9/includes/WebResponse.php on line 38
[23:17:27] 	 andrewbogott: that might be step one: getting wikitech-static to be sync'd more regularly
[23:17:46] 	 Yep, I'll pester Ryan for the keys so that I can learn more about how it works.
[23:17:50] * greg-g  nods
[23:17:58] 	 do you know where -static lives?
[23:18:11] 	 on a rackspace VM I think
[23:18:48] 	 I thought that's where wikitech was
[23:18:57] 	 nope, wikitech is virt0, in tampa
[23:18:57] * greg-g  isn't sure about the location anymore
[23:18:59] 	 ah
[23:19:03] 	 soon to be virt1000 in eqiad
[23:19:05] * greg-g  nods
[23:19:14] 	 Reedy: yeah,  it's supposed to be done, tracking the error now
[23:20:17] 	 Reedy:  So if I sort out the dependency tree for smw, is it safe to say that no one else at WMF will care?  Or do other things care about the current old version of SRF and Validator?
[23:20:34] 	 andrewbogott: then yeah, if we can get it to be sync'd to static regulary, we (who?) can probably work on getting it to have new code when enwiki gets new code (every thursday).
[23:20:57] 	 but, I was honestly thinking for a while that wikitech should only update for security releases and new tarballs
[23:21:01] 	 andrewbogott: Indeed. I think the current combination was Ryans "known working" versions
[23:21:18] 	 (I might be over cautious there though)
[23:21:30] 	 Reedy:  Wait, my question was either/or, if you answer 'yes' I don't know what you're saying...
[23:21:38] 	 haha
[23:22:12] 	 greg-g: There's a lot of active development on OpenStackManager.  I'd prefer for that work to target the latest core rather than an arbitrary old branch.
[23:22:19] 	 Hence, needing to upgrade core pretty often.
[23:22:27] * greg-g  nods
[23:22:32] 	 agree, I was too cautious
[23:22:37] 	 bsitu: findRootPath can return null. Line 243 uses the return value without checking it
[23:22:41] 	 Today's problem was because someone else merged a patch to OSM that only worked with the latest core.  So in order to keep working I had to upgrade everything.
[23:22:46] * greg-g  nods
[23:22:59] 	 Presumably I should've scheduled that someplace…  *sheepish*
[23:23:23] 	 Pretend you know nothing and be shot when it breaks
[23:23:41] 	 andrewbogott: yeah, that'd be really really really good :)
[23:23:55] 	 (03CR) 10Reedy: [C: 032] All non wikipedias to 1.23wmf9 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106149 (owner: 10Reedy)
[23:24:05] 	 (03Merged) 10jenkins-bot: All non wikipedias to 1.23wmf9 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106149 (owner: 10Reedy)
[23:24:08] 	 here we go again
[23:24:16] 	 greg-g:  So, where/how would I do that?
[23:24:27] 	 andrewbogott: tell me :)
[23:24:37] 	 Oh, ok, that's simple enough
[23:24:57] 	 ideally, add it to https://wikitech.wikimedia.org/wiki/Deployments#Next_month
[23:25:19] 	 andrewbogott: basically, read the email "getting things deployed" I sent to engineering@ :)
[23:26:10] 	 Hm… given that that was sent yesterday I don't feel so bad about not following the process therein
[23:26:16] 	 But, reading now!
[23:26:28] 	 And, sorry for the commotion.
[23:27:25] 	 andrewbogott: well, no warning wasn't ok before yesterday either :)
[23:27:32] 	 there's a reason I'm in the weekly ops meeting :)
[23:27:37] 	 reasonable
[23:27:39] 	 and ask "what's coming from your side"
[23:28:10] 	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias back to 1.23wmf9
[23:28:16] 	 Logged the message, Master
[23:28:17] 	 Reedy: yeah, but it shouldn't be null.  An exception should be thrown if null
[23:28:39] 	 (03PS1) 10BryanDavis: [WIP] Add logstash config for udp2log [operations/puppet] - 10https://gerrit.wikimedia.org/r/106154 
[23:30:16] 	 bsitu: Just noticed a couple of these too
[23:30:17] 	 2014-01-07 23:28:35 mw1023 mediawikiwiki: [bb326c4c] /w/index.php?title=Talk:Sandbox&workflow=050b9046b262b234bf36842b2b774df2&action=new-topic   Exception from line 498 of /usr/local/apache/common-local/php-1.23wmf9/extensions/Flow/includes/Data/ObjectManager.php: failed insert
[23:30:44] 	 wikidata is good.... creating items
[23:30:57] 	 Only obvious noise in the exception logs is Flow as above
[23:31:26] 	 jkrauska-mbp:~ jkrauska$ ping6 google.com
[23:31:26] 	 PING6(56=40+8+8 bytes) 2620:62:c000:149:54e0:6baa:291b:62 --> 2607:f8b0:4005:800::1001
[23:31:27] 	 16 bytes from 2607:f8b0:4005:800::1001, icmp_seq=0 hlim=54 time=10.882 ms
[23:31:27] 	 yeah, something is wrong, we are debugging it now
[23:31:28] 	 16 bytes from 2607:f8b0:4005:800::1001, icmp_seq=1 hlim=54 time=12.637 ms
[23:32:36] * Reedy  wonders if cajoel is lost
[23:32:44] 	 just spreading the good cheer
[23:32:47] 	 ipv6 in the office
[23:32:51] 	 Reedy: just being excited.
[23:32:59] 	 wandering around the v6 internet
[23:33:05] 	 not all those who wander are lost
[23:33:11] 	 ahh
[23:33:32] 	 cajoel: is it bigger than you imagined?
[23:33:38] 	 ipv6 in the office is cool
[23:34:00] 	 hello
[23:34:08] 	 greg-g: All looks fine bar Flow now
[23:34:30] 	 blugh
[23:34:36] 	 Reedy: including category tree?
[23:34:37] * greg-g  looks
[23:34:39] 	 Oh, hah
[23:34:41] 	 No idea
[23:34:51] 	 wahoa
[23:34:56] 	 yeah, fixed
[23:35:04] 	 Some stupid js related inconsistency then
[23:35:10] 	 https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=MediaWiki+errors&vl=errors+%2F+sec&x=0.5&n=&hreg[]=vanadium.eqiad.wmnet&mreg[]=fatal|exception>ype=stack&glegend=show&aggregate=1&embed=1
[23:35:17] 	 ^ Nothing scary appearing there either
[23:37:22] 	 Flow's just killin' it lately.
[23:38:13] * Jasper_Deng  wonders if Gloria is talking about the disaster from last night
[23:38:52] 	 Jasper_Deng: why stop at 'disaster'? how about 'catastrophe' or 'apocalypse'?
[23:39:25] 	 at least it was mw.org, enwiki would be up with pitchforks if it happened there
[23:39:29] 	 so not quite an apocalypse
[23:39:45] * Jasper_Deng  thanks ori for the quick end he put to it
[23:40:00] 	 ori: Let's deploy unstable code to the cluster that doesn't include any anti-abuse features.
[23:40:03] 	 i didn't do anything
[23:40:10] * ori  abuses Gloria.
[23:41:03] 	 Has ori got a new stabbing thing?
[23:41:08] 	 (03PS1) 10RobH: replace wildcard cert for contacts.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/106157 
[23:41:54] 	 Jasper_Deng: i'm sure its completely a coincidence that flow is only deployed to mw.org
[23:42:12] 	 Jasper_Deng: to 3 pages
[23:42:42] 	 (03CR) 10RobH: [C: 032] replace wildcard cert for contacts.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/106157 (owner: 10RobH)
[23:42:43] 	 Reedy: he needed it so he could kill the spambots that were attacking last night
[23:42:44] 	 :)
[23:42:51] 	 ebernhardson: heh
[23:43:43] 	 ebernhardson: I fail to see how that's relevant.
[23:43:51] 	 Reedy: just 5 o'clock stubble
[23:43:52] 	 Gloria: then you fail
[23:43:56] 	 Gloria: no suprise :)
[23:44:00] 	 !log reedy updated /a/common to {{Gerrit|I0e19ec875}}: All non wikipedias to 1.23wmf9
[23:44:05] 	 (03PS1) 10Reedy: $wgMemoryLimit up to 220MB [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106158 
[23:44:05] 	 ebernhardson: I fail or Flow fails? Coin toss!
[23:44:07] 	 Logged the message, Master
[23:44:25] 	 Gloria: flow is on track with its plans, so that just leaves you
[23:45:16] 	 ebernhardson: What's the expression? "Gang aft agley"?
[23:45:50] 	 Ganglia AFT5
[23:57:54] 	 (03PS1) 10Reedy: Use uca-cy collation on Welsh projects [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106163