[00:00:20] (03PS1) 10Reedy: Revert "Fixup docroot code to work for wikimanias all from one docroot folder" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84681 [00:02:13] RobH: Sep 18 00:01:37 carbon atftpd[10345]: Serving precise-installer/ubuntu-installer/amd64/initrd.gz to 198.35.26.5:49155 [00:02:16] fingers crossed... [00:02:16] RECOVERY - mysqld processes on db1036 is OK: PROCS OK: 1 process with command name mysqld [00:09:56] hzaehazeazeaze [00:09:59] ether pad died :/ [00:10:04] 500 [00:10:13] it happens every 10 min or so :P [00:10:22] just keep refreshing :D [00:10:41] hm or does that work only for 503 [00:13:52] it is not even monitored ah [00:13:55] well RT time :-] [00:17:22] hrm…. Sep 18 00:17:05 carbon atftpd[11012]: timeout: retrying... [00:17:23] Sep 18 00:17:10 carbon atftpd[11012]: last message repeated 17 times [00:17:30] wondering if the latency on the link is too large [00:18:12] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 304 seconds [00:19:33] (03PS1) 10Springle: return db1050 to the pool [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84686 [00:20:12] (03CR) 10Springle: [C: 032 V: 032] return db1050 to the pool [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84686 (owner: 10Springle) [00:20:59] !log springle synchronized wmf-config/db-eqiad.php [00:21:02] Logged the message, Master [00:23:34] (03PS1) 10Cmjohnson: adding fwd dns for vl1001-1003 [operations/dns] - 10https://gerrit.wikimedia.org/r/84690 [00:24:46] hasharBusy: transposed ticket # [00:25:10] jeremyb: thank you :) [00:25:40] (03CR) 10Cmjohnson: [C: 032] adding fwd dns for vl1001-1003 [operations/dns] - 10https://gerrit.wikimedia.org/r/84690 (owner: 10Cmjohnson) [00:29:58] !log returned db1050 to the pool [00:30:02] Logged the message, Master [00:30:26] !log upgrade, reboot, mariadb db1033 [00:30:29] Logged the message, Master [00:32:37] !log dns update [00:32:41] Logged the message, Master [00:32:48] !log mwalker synchronized php-1.22wmf16/extensions/CentralNotice 'Updating CentralNotice to master' [00:32:51] Logged the message, Master [00:33:01] PROBLEM - Host db1033 is DOWN: CRITICAL - Plugin timed out after 15 seconds [00:33:21] !log mwalker synchronized php-1.22wmf17/extensions/CentralNotice 'Updating CentralNotice to master' [00:33:26] Logged the message, Master [00:35:36] (03PS2) 10Reedy: Revert "Fixup docroot code to work for wikimanias all from one docroot folder" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84681 [00:35:41] (03CR) 10Reedy: [C: 032] Revert "Fixup docroot code to work for wikimanias all from one docroot folder" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84681 (owner: 10Reedy) [00:36:05] (03PS1) 10Reedy: Revert "Simplify wikimania apache conf" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84694 [00:36:39] (03CR) 10jenkins-bot: [V: 04-1] Revert "Fixup docroot code to work for wikimanias all from one docroot folder" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84681 (owner: 10Reedy) [00:37:01] (03CR) 10Reedy: [V: 032] Revert "Fixup docroot code to work for wikimanias all from one docroot folder" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84681 (owner: 10Reedy) [00:37:39] !log reedy synchronized multiversion/MWMultiVersion.php [00:37:47] (03Abandoned) 10Reedy: Remove wikimania year specific docroots [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84561 (owner: 10Reedy) [00:38:04] TimStarling: Am I right in thinking you had some idea about cleaning up our docroot mess? [00:38:37] how do you mean mess? [00:38:54] you can reuse docroots for multiple virtualhosts [00:39:00] I know [00:39:10] yes, I wanted to merge all the document roots and get rid of the use of $_SERVER['DOCUMENT_ROOT'] for wiki detection [00:39:10] i only briefly looked at the patch [00:39:30] But then we have individual ones like wikimania2005, wikimania2006...wikimania2014 [00:39:38] "all"? [00:39:44] what about stuff like favicons? [00:39:46] well, all the wikis [00:39:49] Right, which is what has been annoying mutante and I today [00:39:56] favicons are done by a script already [00:40:05] MW streams them out [00:40:44] If you set ServerAlias to *.wikipedia.org and ServerName to wikipedia, by the time you get it ServerName it is en.wikipedia.org [00:41:10] But if you do ServerName to wikimania and then ServerAlias wikimania2005.wikimedia.org wikimania2006.wikimedia.org etc, you still get a servername of wikimania [00:41:12] but I think apple-touch-icon.png still needs to be handled somehow [00:41:31] Oh [00:41:32] arr, well.. we tried .. [00:41:34] Wait, is that UseCanonicalName off ? [00:41:36] kill the apple devs with fire? [00:41:39] just use $_SERVER['HTTP_HOST'] [00:41:40] so reverting the "simplify wikimania" change [00:41:46] ? [00:42:05] there's no way to make the server name be something useful [00:42:31] "With UseCanonicalName On Apache will use the hostname and port specified in the ServerName directive to construct the canonical name for the server" [00:43:47] (03CR) 10Dzahn: [C: 032] "per talk in office (for now)" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84694 (owner: 10Reedy) [00:43:51] A mixture of $servername (when it's like wikimania) and the docroot is pretty useless [00:44:40] "UseCanonicalName Off" would just make SERVER_NAME be the same as HTTP_HOST, right? [00:44:58] > See also https://gerrit.wikimedia.org/r/51595 and RT# 864 (aka https://bugzilla.wikimedia.org/21919 ) which all seem to prefer docroot verification rather than DNS. [00:45:16] which would seem to defeat the purpose of having it [00:46:15] jeremyb: what's your point? [00:46:17] jeremyb: What's your point? [00:46:20] haha [00:46:37] the wikivoyage vhost has UseCanonicalName On [00:46:43] inconsistent capitalization. but i'll let it go because you're not sitting next to eachother [00:46:59] Reedy: re merging docroots or keeping separate ones [00:46:59] you can just drop those files into the shared document root, since they have a unique filename [00:47:15] !log sync-apache, repool mw1044 [00:47:18] Logged the message, Master [00:47:21] we're not using the wikimania.org domain or anything [00:47:28] in which case, that would've already allievated the problem [00:47:33] Reedy: merged, synced for now. gotta go for the moment [00:47:39] Thanks for your help [00:47:50] yw, let's talk again tomorrow [00:49:49] I'll pick this up again in a bit [00:53:54] PROBLEM - Host db1023 is DOWN: PING CRITICAL - Packet loss = 100% [00:54:31] I have a question, is it technically possible to have a usergroup that can only access deleted files and not have any other tools? [00:55:24] RECOVERY - Host db1023 is UP: PING OK - Packet loss = 0%, RTA = 0.67 ms [00:56:37] Is image stuff seperate from text? [00:56:51] deletedhistory and deletedtext [01:02:03] (03PS1) 10Springle: add db1023 to s2 during shard mariadb upgrade [operations/puppet] - 10https://gerrit.wikimedia.org/r/84700 [01:05:30] (03CR) 10Springle: [C: 032 V: 032] add db1023 to s2 during shard mariadb upgrade [operations/puppet] - 10https://gerrit.wikimedia.org/r/84700 (owner: 10Springle) [01:26:20] PROBLEM - LVS HTTP IPv4 on m.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 21527 bytes in 0.002 second response time [01:29:52] uhh, what pattern do you look for? [01:30:46] did someone kill the old nagios? i forget the name. we should unbreak the redirect to icinga [01:31:32] no [01:31:34] we can't [01:31:45] at all [01:31:46] never [01:32:05] thanks Reedy :) [01:34:25] did we used to be able to get some limited config in web UI? maybe that was nagios [01:35:26] ok, i think i found it [01:36:23] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [01:37:26] i'm unclear why m.wikimedia exists. content seems to be the same as wikimedia.org [01:37:32] Elsie: ^ ? [01:38:08] Elsie: Stop creating unused subdomains [01:39:30] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:40:31] petridge farms remembers when m.* had to go through a ruby proxy >.> [01:40:49] oh, ruby, i remember you! [01:47:06] PROBLEM - mysqld processes on db1023 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [01:48:14] i wonder if legal's still around at this hour [01:53:21] sloooow clone [01:56:33] jeremyb: There's a m.wikimedia? [01:56:38] an, maybe. [01:56:47] An emmmmm. [01:57:07] there is. but turns out that's irrelevant [01:57:20] and actually faidon already patched this. why isn't it fixed yet? [01:57:27] Crack the whip. [01:58:21] Elsie: it's a little confusing because icinga says "m.wikimedia.org" but it's really checking "en.m.wikipedia.org" [01:59:27] I didn't realize anyone listened to icinga. [02:04:40] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [02:07:49] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:09:25] (03PS2) 10Reedy: Change Echo enabling to be based on echowikis.dblist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84636 [02:09:31] (03CR) 10Reedy: [C: 032] Change Echo enabling to be based on echowikis.dblist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84636 (owner: 10Reedy) [02:11:02] (03Merged) 10jenkins-bot: Change Echo enabling to be based on echowikis.dblist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84636 (owner: 10Reedy) [02:11:24] (03PS1) 10Reedy: Add echowikis dblist to noc conf [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84705 [02:11:47] (03PS2) 10Reedy: Add echowikis dblist to noc conf [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84705 [02:11:53] (03CR) 10Reedy: [C: 032] Add echowikis dblist to noc conf [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84705 (owner: 10Reedy) [02:12:04] PROBLEM - MySQL Processlist on db1051 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 37 statistics [02:12:04] (03Merged) 10jenkins-bot: Add echowikis dblist to noc conf [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84705 (owner: 10Reedy) [02:13:08] RECOVERY - MySQL Processlist on db1051 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics [02:13:26] !log reedy synchronized docroot/noc/ [02:13:31] Logged the message, Master [02:14:08] !log reedy synchronized database lists files: [02:14:16] Logged the message, Master [02:14:59] !log reedy synchronized wmf-config/ 'Switch Echo to using dblist for enablement criteria' [02:15:04] Logged the message, Master [02:15:07] !log LocalisationUpdate completed (1.22wmf17) at Wed Sep 18 02:15:07 UTC 2013 [02:15:11] Logged the message, Master [02:16:02] clones slow for all or just me? [02:16:18] at first i thought just me. then i did a speed test and my connection seems ok [02:17:04] seems fine cloning on a vm at home [02:17:43] huh. i was getting ~25-50kbps [02:22:06] it's the goverment jeremyb [02:24:38] NSA reading my clone from an already public repo? :) [02:28:38] !log LocalisationUpdate completed (1.22wmf16) at Wed Sep 18 02:28:37 UTC 2013 [02:28:42] Logged the message, Master [02:28:56] TimStarling: Should most of our sites be on the same robots.txt? I notice some dirs don't have one, while most symlink to /apache/common/robots.txt, there's a handful with their own files [02:31:22] One being wikimaniateam [02:31:29] reedy@tin:/a/common/docroot$ cat wikimaniateam/robots.txt [02:31:29] User-agent: * [02:31:30] Disallow: / [02:32:01] I guess that's an extra safeguard for the "private wikis" [02:33:12] (03PS1) 10Reedy: Simplify wikimania apache conf, reuse wikimedia.org docroot [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84707 [02:33:13] (03CR) 10jenkins-bot: [V: 04-1] Simplify wikimania apache conf, reuse wikimedia.org docroot [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84707 (owner: 10Reedy) [02:36:41] Reedy: robots should normally be served by php iirc. (i guess maybe extract2?) [02:36:56] in most cases it's symlinked into the dirs, but not all [02:37:07] but i don't see the harm in a safety fallback for private wikis [02:41:15] ok, i think i know what happened [02:41:58] but no kaldari at this hour [02:46:19] (03PS1) 10Reedy: Use robots-private.txt symlinked [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84708 [02:46:20] (03PS1) 10Reedy: Condense a few more robot files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84709 [02:49:56] (03PS1) 10Reedy: Remove press docroot, unused [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84710 [02:50:10] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.109 second response time [02:51:32] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 18 02:51:31 UTC 2013 [02:51:35] Logged the message, Master [02:53:25] (03PS1) 10Reedy: Move testwiki favicon to bits [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84711 [02:54:29] (03PS1) 10Reedy: Remove backup incubator favicon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84712 [02:56:07] Reedy: Enable the .ico file type on Commons. :P [02:57:10] (03PS2) 10Reedy: Simplify wikimania apache conf, reuse wikimedia.org docroot [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84707 [02:57:49] (03CR) 10Reedy: [C: 032] Remove backup incubator favicon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84712 (owner: 10Reedy) [02:57:59] (03Merged) 10jenkins-bot: Remove backup incubator favicon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84712 (owner: 10Reedy) [02:58:16] (03PS2) 10Reedy: Move testwiki favicon to bits [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84711 [02:58:21] (03CR) 10Reedy: [C: 032] Move testwiki favicon to bits [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84711 (owner: 10Reedy) [02:58:33] (03Merged) 10jenkins-bot: Move testwiki favicon to bits [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84711 (owner: 10Reedy) [02:59:02] Reedy: v [02:59:03] Reedy: https://code.google.com/p/gerrit/issues/detail?id=1573 [02:59:22] (03PS2) 10Reedy: Remove press docroot, unused [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84710 [02:59:26] (03CR) 10Reedy: [C: 032] Remove press docroot, unused [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84710 (owner: 10Reedy) [02:59:39] (03Merged) 10jenkins-bot: Remove press docroot, unused [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84710 (owner: 10Reedy) [03:00:55] !log reedy synchronized docroot and w [03:00:58] Logged the message, Master [03:04:44] (03CR) 10Reedy: [C: 032] Condense a few more robot files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84709 (owner: 10Reedy) [03:04:47] (03CR) 10jenkins-bot: [V: 04-1] Condense a few more robot files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84709 (owner: 10Reedy) [03:06:54] (03PS2) 10Reedy: Use robots-private.txt symlinked [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84708 [03:07:22] (03CR) 10Reedy: [C: 032] Use robots-private.txt symlinked [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84708 (owner: 10Reedy) [03:07:43] ok, reopened 53595 [03:07:47] (03Merged) 10jenkins-bot: Use robots-private.txt symlinked [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84708 (owner: 10Reedy) [03:07:54] (bug) [03:08:04] (03PS2) 10Reedy: Condense a few more robot files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84709 [03:08:10] (03CR) 10Reedy: [C: 032] Condense a few more robot files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84709 (owner: 10Reedy) [03:08:31] (03Merged) 10jenkins-bot: Condense a few more robot files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84709 (owner: 10Reedy) [03:12:16] Reedy: in all the wiki-related vhosts, robots.txt is aliased to robots.php, so if there is a file there, it has no effect [03:13:00] the alias hides it [03:17:58] So in most of them it serves no purpose [03:21:35] (03CR) 10Bsitu: "Thanks for taking care of this! This makes the code much more maintainable" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84636 (owner: 10Reedy) [03:26:24] So bar bits and www.*... [03:35:23] (03PS1) 10Reedy: Delete all useless robots.txt files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84713 [03:44:02] (03PS3) 10Reedy: Simplify wikimania apache conf, reuse wikimedia.org docroot. [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84707 [03:51:23] (03PS1) 10Reedy: Remove old commented code [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84714 [03:53:57] (03CR) 10Reedy: [C: 032] Delete all useless robots.txt files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84713 (owner: 10Reedy) [03:54:08] (03Merged) 10jenkins-bot: Delete all useless robots.txt files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84713 (owner: 10Reedy) [03:54:52] !log reedy synchronized docroot and w [03:56:10] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [03:59:50] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:03:04] (03PS1) 10Reedy: No need for these robots.txt either (redirect to en project) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84715 [04:03:58] (03CR) 10Reedy: [C: 032] No need for these robots.txt either (redirect to en project) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84715 (owner: 10Reedy) [04:04:08] (03Merged) 10jenkins-bot: No need for these robots.txt either (redirect to en project) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84715 (owner: 10Reedy) [04:05:07] !log reedy synchronized docroot and w [04:05:41] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [04:08:50] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:09:19] (03PS1) 10Reedy: testwiki.ico is a dupe of black-global.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84717 [04:10:01] (03CR) 10Reedy: [C: 032] testwiki.ico is a dupe of black-global.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84717 (owner: 10Reedy) [04:11:06] (03Merged) 10jenkins-bot: testwiki.ico is a dupe of black-global.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84717 (owner: 10Reedy) [04:15:40] (03PS1) 10Reedy: Kill these symlinks too, favicon.php serves them [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84718 [04:15:41] PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 62 statistics [04:16:14] (03CR) 10Reedy: [C: 032] Kill these symlinks too, favicon.php serves them [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84718 (owner: 10Reedy) [04:16:30] (03Merged) 10jenkins-bot: Kill these symlinks too, favicon.php serves them [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84718 (owner: 10Reedy) [04:16:54] RECOVERY - MySQL Processlist on db1052 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 9 statistics [04:18:35] Good. Most of the docroot dirs have no unique files in them [04:31:52] (03PS1) 10Spage: Add Flow extension (in use on beta labs only!) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84719 [04:31:54] (03CR) 10jenkins-bot: [V: 04-1] Add Flow extension (in use on beta labs only!) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84719 (owner: 10Spage) [04:35:58] (03PS4) 10Reedy: Simplify wikimania apache conf, reuse wikimedia.org docroot. [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84707 [04:36:55] (03PS1) 10Reedy: Kill wikimania docroot folders [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84721 [04:41:56] RECOVERY - MySQL Replication Heartbeat on db1046 is OK: OK replication delay 0 seconds [04:47:57] (03CR) 10Spage: "recheck" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84719 (owner: 10Spage) [04:51:49] (03CR) 10Reedy: [C: 04-1] "(1 comment)" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84714 (owner: 10Reedy) [04:52:55] (03PS2) 10Reedy: Remove old commented code [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84714 [05:05:02] (03PS1) 10Reedy: Remove "labs" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84724 [05:05:42] (03PS1) 10Reedy: Kill labs docroot, long dead [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84725 [05:07:31] (03CR) 10Reedy: [C: 032] Kill labs docroot, long dead [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84725 (owner: 10Reedy) [05:07:41] (03Merged) 10jenkins-bot: Kill labs docroot, long dead [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84725 (owner: 10Reedy) [05:10:42] !log reedy synchronized docroot and w [05:22:48] (03PS1) 10Reedy: $wgStockPath is long dead [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84727 [05:23:11] (03CR) 10Reedy: [C: 032] $wgStockPath is long dead [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84727 (owner: 10Reedy) [05:24:25] (03Merged) 10jenkins-bot: $wgStockPath is long dead [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84727 (owner: 10Reedy) [05:26:53] PROBLEM - LVS HTTP IPv4 on m.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 21527 bytes in 0.020 second response time [05:31:06] still [05:31:08] goddammit [05:32:49] lvs upset? [05:33:51] the footer of en.m changed [05:34:05] from "Text is available under" to "Content ..." [05:34:23] we thought we fixed it earlier, kaldari proposed "Last modified on" [05:34:48] but that doesn't appear apparently [05:39:30] (03PS1) 10Faidon Liambotis: Change m.wikipedia.org check pattern again [operations/puppet] - 10https://gerrit.wikimedia.org/r/84728 [05:39:57] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Change m.wikipedia.org check pattern again [operations/puppet] - 10https://gerrit.wikimedia.org/r/84728 (owner: 10Faidon Liambotis) [05:40:48] RECOVERY - LVS HTTP IPv4 on m.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 21526 bytes in 0.007 second response time [06:35:49] paravoid: re "Content" https://bugzilla.wikimedia.org/53595#c8 [06:36:56] paravoid: also, it's a bit confusing that the pattern check is for a different host than what icinga complains about. (the check is for en.m.wp.o and the alert is for m.wm.o) [07:31:45] morning [07:42:09] (03CR) 10Hashar: [C: 04-1] "The file is used by MediaWiki core maintenance script maintenance/mergeMessageFileList.php . The '$IP' string is replaced by the $IP glob" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84719 (owner: 10Spage) [07:48:15] PROBLEM - Puppet freshness on hume is CRITICAL: No successful Puppet run in the last 10 hours [07:52:19] hashar: {{fact}} [07:53:15] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours [07:58:25] jeremyb: fact1: slept 2 hours, consequence1: I am tired :-D [09:13:09] PROBLEM - Frontend Squid HTTP on sq45 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:17:19] PROBLEM - SSH on sq45 is CRITICAL: Server answer: [09:33:59] RECOVERY - Frontend Squid HTTP on sq45 is OK: HTTP OK: HTTP/1.0 200 OK - 530 bytes in 0.054 second response time [09:34:18] RECOVERY - SSH on sq45 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [10:03:42] PROBLEM - RAID on labsdb1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:05:51] PROBLEM - mysqld processes on labsdb1003 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [10:53:30] (03PS1) 10QChris: Switch to single Json object for gerrit's reviewer count query [operations/puppet] - 10https://gerrit.wikimedia.org/r/84743 [10:55:54] (03CR) 10QChris: [C: 04-1] "Marking as -1, as we need to upgrade gerrit to contain" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84743 (owner: 10QChris) [11:22:59] PROBLEM - Disk space on labsdb1003 is CRITICAL: DISK CRITICAL - /a is not accessible: Input/output error [14:47:58] (03PS1) 10Petr Onderka: Handle progress reporting even when tellg() doesn't work [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84748 [15:26:11] (03PS2) 10Petr Onderka: Handle progress reporting even when tellg() doesn't work [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84748 [15:50:33] (03CR) 10Petr Onderka: [C: 032 V: 032] Handle progress reporting even when tellg() doesn't work [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84748 (owner: 10Petr Onderka) [16:07:26] (03PS1) 10Petr Onderka: Command like parameter for specifying frequency of progress reporting for idumps create [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84759 [16:09:32] (03CR) 10Petr Onderka: [C: 032 V: 032] Command like parameter for specifying frequency of progress reporting for idumps create [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84759 (owner: 10Petr Onderka) [16:15:05] (03PS1) 10Dr0ptp4kt: Add Beeline Kazakhstan for baselining [operations/puppet] - 10https://gerrit.wikimedia.org/r/84760 [16:16:09] (03CR) 10Krinkle: [C: 04-1] "Afaik there is no need for beta extensions to be listed here. I'm not sure how it has been done in the past, but afaik we've installed new" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84719 (owner: 10Spage) [16:16:22] (03CR) 10Dr0ptp4kt: "This can be implemented immediately upon review and approval." [operations/puppet] - 10https://gerrit.wikimedia.org/r/84760 (owner: 10Dr0ptp4kt) [16:18:31] paravoid, request for review and approval of https://gerrit.wikimedia.org/r/#/c/84760/, please (customary zero vcl acl) [17:02:38] (03CR) 10Chad: [C: 032] More StartProfiler.php cleanup [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80528 (owner: 10PleaseStand) [17:04:30] (03Merged) 10jenkins-bot: More StartProfiler.php cleanup [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/80528 (owner: 10PleaseStand) [17:12:25] (03CR) 10Faidon Liambotis: [C: 032] Add Beeline Kazakhstan for baselining [operations/puppet] - 10https://gerrit.wikimedia.org/r/84760 (owner: 10Dr0ptp4kt) [17:14:13] (03PS1) 10RobH: ulsfo & esams subnets addition to apt squid config [operations/puppet] - 10https://gerrit.wikimedia.org/r/84769 [17:15:49] RobH: just use the entire subnet instead of /28 [17:15:57] RobH: should be safe [17:20:04] (03CR) 10RobH: [C: 032] ulsfo & esams subnets addition to apt squid config [operations/puppet] - 10https://gerrit.wikimedia.org/r/84769 (owner: 10RobH) [17:20:49] akosiaris: when you got kafka to build this last time, did you remove the 2.9 or the 2.8 Annotations file? [17:21:38] 2.8 but that was on a hunch, since we got scala 2.9.1. It might work by removing any of the two files [17:21:49] (03PS2) 10BBlack: Add netmapper as git submodule + autotools patch [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84639 [17:25:06] (03CR) 10Akosiaris: [C: 032] contint: publish Zuul git over git protocol [operations/puppet] - 10https://gerrit.wikimedia.org/r/82625 (owner: 10Hashar) [17:35:01] (03Abandoned) 10Ladsgroup: Enabling Echo and Thanks in Persian Wikipedia Bug: 54068 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84203 (owner: 10Ladsgroup) [17:37:22] greg-g: Good morning, it's another great day for a lightning deploy request :) the fixes I wanted to deploy yesterday came through today, so can I get another LD slot today to deploy those? [17:38:22] PROBLEM - MySQL Slave Running on db1001 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Error on rename of ./puppet/#sql-2b5_2 to ./puppet/hwdoc_eq [17:41:10] RoanKattouw: :) yessir [17:42:12] PROBLEM - Backend Squid HTTP on sq45 is CRITICAL: Connection refused [17:42:18] Awesome, thanks [17:42:52] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [17:43:33] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 0.93 ms [17:45:09] https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=83819&oldid=83789 [17:46:02] PROBLEM - Apache HTTP on mw1085 is CRITICAL: Connection refused [17:48:22] PROBLEM - Puppet freshness on hume is CRITICAL: No successful Puppet run in the last 10 hours [17:48:58] (03CR) 10Chad: [C: 031] Break lines in review Gerrit comments [operations/puppet] - 10https://gerrit.wikimedia.org/r/84522 (owner: 10Dereckson) [17:53:22] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours [17:59:02] RECOVERY - Apache HTTP on mw1085 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 3.175 second response time [18:05:45] ori-l: I'm sitting at your desk waiting for you :P [18:05:49] well "sitting" [18:14:31] !log reedy synchronized php-1.22wmf17/extensions/TimedMediaHandler [18:14:34] Logged the message, Master [18:15:13] (03PS3) 10BBlack: Add netmapper as full debian patch [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84639 [18:18:45] (03PS1) 10Reedy: Drop Video Scalers to 5 concurrent jobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/84775 [18:19:23] ^ Can someone review that please (ie confirm that's doing what I thikn it's doing)? [18:20:42] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [18:21:17] greg-g: Can I deploy CentralAuth now? I'm assuming zero is done? [18:21:56] csteipp: I assume so as well, yeah, go ahead [18:23:53] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:37:22] PROBLEM - Puppet freshness on sq45 is CRITICAL: No successful Puppet run in the last 10 hours [18:38:10] (03CR) 10Akosiaris: [C: 032] Drop Video Scalers to 5 concurrent jobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/84775 (owner: 10Reedy) [18:39:15] <^d> Ryan_Lane: So I was working on iptables rules for nrpe on ytterbium. I got this far from copy+pasting, but I really have nfc what I'm doing. http://p.defau.lt/?V1x8Kkz70m9m5_hR0fweMQ [18:40:04] we should use ferm [18:40:14] (03PS1) 10Reedy: wmgHTTPSBlacklistCountries changes for CN [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84781 [18:40:42] <^d> Ryan_Lane: I'm fine with using avian carriers if it means we can get monitoring in place for this. I'm kinda tired of waiting on things I don't understand :) [18:40:51] :D [18:40:55] well, I don't know how to use ferm either [18:41:05] but there's support for it in puppet now [18:41:13] and the old iptables stuff is just terrible [18:41:20] wait. is this just a change in the current rules? [18:41:31] or a whole new set? [18:41:38] <^d> No, we have no iptables rules in puppet for gerrit. [18:41:42] <^d> Never have afaik. [18:41:46] ok, yeah, we should use ferm, then [18:42:37] ferm ftw! [18:43:17] <^d> Ok so how do we do this? :) [18:43:22] no clue :) [18:43:31] let me see what's there [18:44:01] !log csteipp synchronized php-1.22wmf17/extensions/CentralAuth 'update centralauth to master for SUL fixes' [18:44:05] Logged the message, Master [18:44:33] !log csteipp synchronized php-1.22wmf16/extensions/CentralAuth 'update centralauth to master for SUL fixes' [18:44:37] Logged the message, Master [18:46:36] (03PS1) 10BBlack: fix changelog typo [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84782 [18:46:37] (03PS1) 10BBlack: update libvarnishapi1 symbols [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84783 [18:46:38] (03PS1) 10BBlack: varnish (3.0.3plus~rc1-wm15) precise; urgency=low [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84784 [18:48:16] btw, AaronSchulz, were you working on the favicon.ico exception earlier? [18:48:40] (03CR) 10BBlack: [C: 032 V: 032] "Not ideal, but it's what we do for now." [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84639 (owner: 10BBlack) [18:48:53] (03CR) 10BBlack: [C: 032 V: 032] fix changelog typo [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84782 (owner: 10BBlack) [18:49:06] (03CR) 10BBlack: [C: 032 V: 032] update libvarnishapi1 symbols [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84783 (owner: 10BBlack) [18:49:44] (03CR) 10BBlack: [C: 032 V: 032] varnish (3.0.3plus~rc1-wm15) precise; urgency=low [operations/debs/varnish] (testing/3.0.3plus-rc1) - 10https://gerrit.wikimedia.org/r/84784 (owner: 10BBlack) [18:51:05] (03PS1) 10Hashar: contint: typo packages {} -> package {} [operations/puppet] - 10https://gerrit.wikimedia.org/r/84785 [18:53:34] (03CR) 10Akosiaris: [C: 032] contint: typo packages {} -> package {} [operations/puppet] - 10https://gerrit.wikimedia.org/r/84785 (owner: 10Hashar) [18:55:01] (03CR) 10Hashar: "packages {} -> package {} typo fixed in https://gerrit.wikimedia.org/r/84785 . Sorry :(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/82625 (owner: 10Hashar) [18:55:16] What's the dbname for the affcom wiki? [18:55:33] chapcomwiki [18:55:58] thanks [18:57:22] (03PS1) 10Reedy: Enable Collection on chapcomwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84786 [18:57:43] (03CR) 10Reedy: [C: 032] Enable Collection on chapcomwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84786 (owner: 10Reedy) [18:58:21] (03Merged) 10jenkins-bot: Enable Collection on chapcomwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84786 (owner: 10Reedy) [19:06:32] pooor stafford [19:08:59] PROBLEM - Host labsdb1003 is DOWN: PING CRITICAL - Packet loss = 100% [19:12:51] !log mwalker synchronized php-1.22wmf16/extensions/CentralNotice/ 'Updating CentralNotice to master' [19:12:54] Logged the message, Master [19:13:20] !log mwalker synchronized php-1.22wmf17/extensions/CentralNotice/ 'Updating CentralNotice to master' [19:13:22] hashar: I have no idea what's going on with zuul/jenkins today -- but it is VERY slow running on core [19:13:24] Logged the message, Master [19:14:00] mwalker: yeah bunch of tests running ( https://integration.wikimedia.org/zuul/ ) [19:14:10] RECOVERY - Host labsdb1003 is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms [19:14:14] mwalker: I will hopefully manage to get the second slave in very soon [19:14:21] its more than that -- it still hasn't picked up my 84790 job [19:15:56] mwalker: basically gallium is starving for disk I/O :-( [19:16:08] even with an ssd? [19:16:10] impressive [19:16:10] PROBLEM - SSH on labsdb1003 is CRITICAL: Connection refused [19:16:11] PROBLEM - MySQL Idle Transactions Port 3307 on labsdb1003 is CRITICAL: Connection refused by host [19:16:11] PROBLEM - MySQL Recent Restart Port 3308 on labsdb1003 is CRITICAL: Connection refused by host [19:16:11] PROBLEM - MySQL Slave Delay Port 3306 on labsdb1003 is CRITICAL: Connection refused by host [19:16:20] PROBLEM - MySQL Slave Delay Port 3307 on labsdb1003 is CRITICAL: Connection refused by host [19:16:20] PROBLEM - MySQL Slave Delay Port 3308 on labsdb1003 is CRITICAL: Connection refused by host [19:16:30] PROBLEM - MySQL Slave Running Port 3308 on labsdb1003 is CRITICAL: Connection refused by host [19:16:30] PROBLEM - MySQL Slave Running Port 3307 on labsdb1003 is CRITICAL: Connection refused by host [19:16:40] PROBLEM - MySQL Idle Transactions Port 3306 on labsdb1003 is CRITICAL: Connection refused by host [19:16:40] PROBLEM - DPKG on labsdb1003 is CRITICAL: Connection refused by host [19:16:40] PROBLEM - MySQL Slave Running Port 3306 on labsdb1003 is CRITICAL: Connection refused by host [19:16:50] PROBLEM - MySQL Idle Transactions Port 3308 on labsdb1003 is CRITICAL: Connection refused by host [19:16:50] PROBLEM - MySQL Recent Restart Port 3306 on labsdb1003 is CRITICAL: Connection refused by host [19:16:50] PROBLEM - RAID on labsdb1003 is CRITICAL: Connection refused by host [19:17:00] PROBLEM - MySQL Recent Restart Port 3307 on labsdb1003 is CRITICAL: Connection refused by host [19:26:00] PROBLEM - Host labsdb1003 is DOWN: PING CRITICAL - Packet loss = 100% [19:31:26] oooh, I forgot SuchABot doesn't watch all our repos on GitHub [19:31:48] csteipp: never heard of it [19:36:05] LeslieCarr: what did you do in throttle.php yesterday? something changed that caused it to start spamming the exception logs [19:36:28] (but strangely the code that is throwing the exception is code that hasnt changed since late 2012; so...) [19:36:30] RECOVERY - Host labsdb1003 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [19:37:08] RECOVERY - MySQL Idle Transactions Port 3308 on labsdb1003 is OK: OK longest blocking idle transaction sleeps for seconds [19:37:08] RECOVERY - SSH on labsdb1003 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [19:37:08] RECOVERY - MySQL Idle Transactions Port 3307 on labsdb1003 is OK: OK longest blocking idle transaction sleeps for seconds [19:37:08] RECOVERY - MySQL Recent Restart Port 3308 on labsdb1003 is OK: OK seconds since restart [19:37:18] RECOVERY - MySQL Recent Restart Port 3306 on labsdb1003 is OK: OK seconds since restart [19:37:18] RECOVERY - MySQL Slave Delay Port 3306 on labsdb1003 is OK: OK replication delay 0 seconds [19:37:18] RECOVERY - MySQL Slave Delay Port 3307 on labsdb1003 is OK: OK replication delay 0 seconds [19:37:18] RECOVERY - MySQL Slave Delay Port 3308 on labsdb1003 is OK: OK replication delay 0 seconds [19:37:58] RECOVERY - MySQL Recent Restart Port 3307 on labsdb1003 is OK: OK seconds since restart [19:38:48] RECOVERY - Disk space on labsdb1003 is OK: DISK OK [19:42:08] RECOVERY - mysqld processes on labsdb1003 is OK: PROCS OK: 3 processes with command name mysqld [19:42:40] mwalker: it just changed an ip [19:42:45] and date [19:42:53] greg-g, i missed depl window, but would still like to push it out today (syncdiring zero, shouldn't be too demaging :)). Should I wait for lighting deploy or can i go earlier so there is more time to fix issues :) [19:42:56] think it could be from the deploying ? [19:43:07] incubatorwiki ApiWikiLoveImageLog::saveInDb 10.64.16.8 1146 Table 'incubatorwiki.wikilove_image_log' doesn't exist (10.64.16.8) [19:43:13] * AaronSchulz sighs [19:43:17] mwalker, are you guys deploying? [19:44:52] LeslieCarr: I thought we were -8? [19:44:57] * AaronSchulz must have misremembered [19:45:00] daylight savings time [19:45:59] LeslieCarr: :) [19:51:31] !log stopping mysqld instances on labsdb1003 for troubleshooting after xfs crash/corruption [19:51:34] Logged the message, Master [19:53:08] PROBLEM - mysqld processes on labsdb1003 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [20:02:25] (03PS2) 10Nikerabbit: ULS eventlogging phase 2 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84355 [20:03:59] (03CR) 10Ladsgroup: [C: 031] "Thank you so much." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [20:04:34] !log Jenkins: migrated sartoris jobs to fetch Zuul reference over the git protocol made available with {{bug|53683}}. Test change is https://gerrit.wikimedia.org/r/#/c/46450/ [20:04:38] Logged the message, Master [20:06:58] (03CR) 10Nikerabbit: [C: 032] ULS eventlogging phase 2 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84355 (owner: 10Nikerabbit) [20:07:12] (03Merged) 10jenkins-bot: ULS eventlogging phase 2 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84355 (owner: 10Nikerabbit) [20:07:30] (03PS1) 10Cmjohnson: adding ebernhardson to site.pp/admins.pp rt5717 [operations/puppet] - 10https://gerrit.wikimedia.org/r/84854 [20:10:45] !log nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS EventLogging' [20:10:48] Logged the message, Master [20:11:00] (03CR) 10Cmjohnson: [C: 031] adding ebernhardson to site.pp/admins.pp rt5717 [operations/puppet] - 10https://gerrit.wikimedia.org/r/84854 (owner: 10Cmjohnson) [20:11:35] (03CR) 10RobH: [C: 04-1] "needs deployment, so add to mortals as well as to specific stat server" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84854 (owner: 10Cmjohnson) [20:12:19] (03PS1) 10Nikerabbit: ULS EventLogging phase 3 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84856 [20:13:04] yuri_k: apologies -- wandered away; if its still relevant -- yes I'm well done [20:13:20] LeslieCarr: it could be guess it sort of depends on when the last time it was synced was [20:13:25] yuri_k: 3pm is open [20:14:41] (03CR) 10Nikerabbit: [C: 032] ULS EventLogging phase 3 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84856 (owner: 10Nikerabbit) [20:14:53] (03Merged) 10jenkins-bot: ULS EventLogging phase 3 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84856 (owner: 10Nikerabbit) [20:15:47] !log nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS EventLogging' [20:17:00] (03PS2) 10Cmjohnson: adding ebernhardson to site.pp/admins.pp rt5717 [operations/puppet] - 10https://gerrit.wikimedia.org/r/84854 [20:31:17] (03CR) 10Cmjohnson: [C: 031] adding ebernhardson to site.pp/admins.pp rt5717 [operations/puppet] - 10https://gerrit.wikimedia.org/r/84854 (owner: 10Cmjohnson) [20:32:18] (03CR) 10RobH: [C: 031] adding ebernhardson to site.pp/admins.pp rt5717 [operations/puppet] - 10https://gerrit.wikimedia.org/r/84854 (owner: 10Cmjohnson) [20:32:45] robh ^ [20:32:56] ? [20:32:58] i just reviewed it. [20:34:09] (03CR) 10Cmjohnson: [C: 032] adding ebernhardson to site.pp/admins.pp rt5717 [operations/puppet] - 10https://gerrit.wikimedia.org/r/84854 (owner: 10Cmjohnson) [20:38:37] !log cloning from sanitarium to labsdb1003 instances [20:38:40] Logged the message, Master [20:49:46] (03PS2) 10Reedy: Set $wgFFmpegThreads to 2 for faster transcodes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84539 (owner: 10J) [20:49:54] (03CR) 10Reedy: [C: 032] Set $wgFFmpegThreads to 2 for faster transcodes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84539 (owner: 10J) [20:50:07] (03Merged) 10jenkins-bot: Set $wgFFmpegThreads to 2 for faster transcodes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84539 (owner: 10J) [20:50:38] (03PS2) 10Reedy: Add Cookbook and Wikijunior namespaces for hywikibooks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84501 (owner: 10TTO) [20:50:42] (03CR) 10Reedy: [C: 032] Add Cookbook and Wikijunior namespaces for hywikibooks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84501 (owner: 10TTO) [20:50:59] (03Merged) 10jenkins-bot: Add Cookbook and Wikijunior namespaces for hywikibooks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84501 (owner: 10TTO) [20:51:12] oh, one of Reedy famous merge sprees! popcorn time! [20:51:39] There's not that many [20:52:02] (03PS5) 10Reedy: Add patrolmarks right to autoconfirmed user group on ckb.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [20:52:08] (03CR) 10Reedy: [C: 032] Add patrolmarks right to autoconfirmed user group on ckb.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [20:52:17] There's a handful to do [20:52:22] (03Merged) 10jenkins-bot: Add patrolmarks right to autoconfirmed user group on ckb.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84206 (owner: 10Ladsgroup) [20:52:32] Still a great job, Reedy sir. [20:52:54] (03PS2) 10Reedy: PNG instead of SVG rendered for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84205 (owner: 10Ebrahim) [20:53:01] (03CR) 10Reedy: [C: 032] PNG instead of SVG rendered for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84205 (owner: 10Ebrahim) [20:53:17] (03Merged) 10jenkins-bot: PNG instead of SVG rendered for Persian Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84205 (owner: 10Ebrahim) [20:54:18] !log reedy synchronized wmf-config/ [20:54:22] Logged the message, Master [20:55:37] Reedy: I assume you don't hear that too often, so once again, great job on all these merges [21:00:52] (03CR) 10Anomie: [C: 031] "Code seems sane. Haven't tested." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/83225 (owner: 10Umherirrender) [21:01:59] greg-g: so my calendar is telling me it is my turn. any ceremony before I start? [21:02:29] manybubbles: run around in a cirlce 3 times, tell your mom you love her, and press enter [21:02:58] done. anyone mind if I upgrade elasticsearch on the cluster now? [21:03:14] I think you're good [21:03:22] I am, then. [21:05:07] manybubbles: have you told your mum you love her before proceeding with deployment ? [21:05:18] hashar: I sent her a text [21:05:25] so you should be fine :-] [21:05:27] good luck ! [21:06:38] !log Created EducationProgram tables on enwikinews [21:06:41] Logged the message, Master [21:07:07] !g project:qa/browsertests owner:self [21:07:08] https://gerrit.wikimedia.org/r/#q,project:qa/browsertests,n,z [21:07:13] almost [21:07:57] (03PS1) 10Reedy: Enable EducationProgram on enwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84863 [21:08:28] greg-g: all done [21:09:03] manybubbles: do you know whether we have any plan to migrate the existing solr instances we have to ElasticSearch ? [21:09:18] now I'm supposed to use the log thing. Looks just like the one in beta but without deployment-prep prefix [21:09:46] hashar: MaxSem has just started looking at it for the geo stuff. [21:09:49] !log manybubbles told his mom he loved her [21:09:53] Logged the message, Master [21:10:01] !log Upgraded elasticsearch to 0.90.4. [21:10:05] Logged the message, Master [21:10:11] (03PS2) 10Reedy: Enable EducationProgram on enwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84863 [21:10:12] !log greg-g showed me how to log stuff [21:10:15] Logged the message, Master [21:10:38] hashar: Nikerabbit also has made a bug to move ttmserver [21:10:58] Uh [21:11:00] hashar: https://gerrit.wikimedia.org/r/#/c/84863/ [21:11:05] Why did patch one work? [21:11:07] I had => tru [21:11:15] manybubbles: fine :) thx! [21:11:18] (03CR) 10Reedy: [C: 032] Enable EducationProgram on enwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84863 (owner: 10Reedy) [21:11:27] Reedy: what is wrong with this change ? [21:11:38] hashar: I wish it were more fast [21:11:46] +» 'enwikinews' => tru, // bug 53984 [21:12:12] that is valid [21:12:14] Notice: Use of undefined constant tru - assumed 'tru' in php shell code on line 1 [21:12:14] :D [21:12:27] I mean, the syntax is valid [21:12:28] ffs php [21:12:33] yeah [21:12:44] the plan is to decouple the GUI and the backend [21:12:59] then we can rewrite the GUI in pure JS and the backend with python :-] [21:14:12] !log Created EducationProgram tables on cswiki [21:14:16] Logged the message, Master [21:14:23] (03Merged) 10jenkins-bot: Enable EducationProgram on enwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84863 (owner: 10Reedy) [21:15:06] !log reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on enwikinews' [21:15:09] Logged the message, Master [21:20:17] hashar: trolling again? [21:22:18] Nikerabbit: hey [21:23:55] manybubbles: yeah mostly :/ [21:24:35] paravoid, should Snaps_ add logging support to varnishkafka [21:24:36] ? [21:25:28] (log to file rather than syslog that is) [21:25:57] (ja) [21:26:54] if you want to [21:27:21] hashar: about yesterday, do you know what the issue is with the messages on beta for Flow? http://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:Flow/Sandbox (and can I help put them in place?) [21:27:41] haven't I replied by email this need a bug report ? :-D [21:27:58] (03CR) 10CSteipp: [C: 031] wmgHTTPSBlacklistCountries changes for CN [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84781 (owner: 10Reedy) [21:28:02] chrismcmahon: S did submit a change. The extension need to be registered in an extension-list file of operations/mediawiki-config.git [21:28:03] hashar: maybe so. :-) [21:28:12] chrismcmahon: and Flow has to be added to the wmf branches in production. [21:28:39] chrismcmahon: then the i18n updater will be able to find the Flow i18n files and add its messages entry to the i18n cache [21:28:44] (yeah I know that is cumbersome) [21:28:57] hashar: I guess I am confused about what it means to be in "in production". [21:29:29] (03PS1) 10Lcarr: updating apt-proxy ip range and tftp ip range [operations/puppet] - 10https://gerrit.wikimedia.org/r/84867 [21:29:50] the extension-list file is shared between production and labs. It list the extensions entry points, if any is missing the script dies horribly. On beta that is not a problem since we have all extension fetched [21:30:09] chrismcmahon: but in production we will be missing the Flow extension since its code is not deployed yet [21:31:48] bblack: Would you happen to know where the change to the upload varnish servers needs to be made to make them gzip svg files? [21:31:49] hashar: this is the conversation? https://gerrit.wikimedia.org/r/#/c/84719/ [21:32:16] https://bugzilla.wikimedia.org/54291 [21:32:59] chrismcmahon: ah yes, that is the change. [21:33:10] Snaps_: paravoid and I are in a big ops meeting, so hard to discuss, don't worry about it for now [21:33:13] paravoid and I will talk later [21:33:15] (03CR) 10Lcarr: [C: 032] updating apt-proxy ip range and tftp ip range [operations/puppet] - 10https://gerrit.wikimedia.org/r/84867 (owner: 10Lcarr) [21:33:27] ottomata: sure, just let me know [21:33:32] hashar: how did we do this for CirrusSearch extension then? maybe the i18n messages don't matter there? [21:33:53] chrismcmahon: no clue :/ [21:34:08] chrismcmahon: probably got added to extension-list since I think it was being tested on test2 as well [21:34:17] RobH: how's the install look ? [21:36:53] hashar: CirrusSearch added to extensions-list on August 15, but it was on beta for weeks before that. [21:37:45] chrismcmahon: CirrusSearch has 3 messages [21:38:07] hmm [21:38:39] One for Special:Version, and the other for search backend errors [21:39:55] Reedy: so the problem is: how do we update the messages for an extension on beta that is not (yet) in production? is that the BZ I should create here? [21:40:50] How've we done it previously? [21:40:53] Or haven't we? [21:41:12] we can potentially tweak the mw-update-l10n script to behave differently when being invoked on beta [21:41:14] <^d> I just kinda deployed to beta. [21:41:17] <^d> And it Just Worked. [21:41:19] Reedy: CirrusSearch is the one that I know of [21:41:23] <^d> Magic! [21:41:44] ^d: probably had missing messages that no one noticed or cared about I'd guess [21:41:53] the trick is that we could use the mergeMessageFile.php maintenance script --extension-dir option, that will automatically detect the entry point [21:42:01] <^d> I don't remember. [21:42:02] and keep using the old extension-list in production [21:42:08] * ^d goes back to debugging things [21:42:35] we need to transclude extension-list into extension-list-labs [21:42:37] and then add stuff [21:43:07] I am not sure we need to exilicitly describe them [21:43:14] Probably not [21:43:18] that is merely listing the extensions entry points isn't it ? [21:43:41] Yeah [21:43:45] But we have stuff like [21:43:45] $IP/extensions/Wikibase/lib/WikibaseLib.php [21:43:45] $IP/extensions/Wikibase/repo/Wikibase.php [21:43:45] $IP/extensions/Wikibase/client/WikibaseClient.php [21:43:52] holy crap :D [21:44:16] <^d> Reedy: That's less the fault of the file and more the fault of how that extension is organized ;-) [21:44:20] springle: Coren: you're reslaving labdb1003 from scratch via hotbackup? :( [21:45:01] Reedy: apparently we can use both the extension-list and the extension-dir options. Though that might introduce duplicate entries [21:45:12] springle: Want to field that one? ^^ [21:45:18] We could almost do with it taking multiple extension-list files [21:45:29] so we can have a supplementary one for beta only stuff [21:45:29] looks like https://bugzilla.wikimedia.org/show_bug.cgi?id=53061 is re-opened for this [21:45:43] * Reedy looks at mergeMessageFileList.php [21:46:49] binasher: yup [21:47:05] I really don't want to just add a list-file2 parameter or similar [21:47:59] list-file=$MW_COMMON_SOURCE/wmf-config/extension-list|$MW_COMMON_SOURCE/wmf-config/extension-list-labs [21:48:00] springle: ah, you saved the old directories… going to restore the user db's, *_p dbs, and grants from the db's under *_bad? [21:48:16] | isn't going to work though then [21:48:30] (03PS1) 10RobH: adding in wikimediastories.com/net/org [operations/dns] - 10https://gerrit.wikimedia.org/r/84872 [21:48:39] binasher: manage_replicas.pl will regenerate the _p dbs; but it's not clear that we /can/ save the user databases. [21:48:41] binasher: yes. Coren says he has sufficient scripts to restore stuff. not sure about user dbs [21:49:04] (03CR) 10RobH: [C: 032] adding in wikimediastories.com/net/org [operations/dns] - 10https://gerrit.wikimedia.org/r/84872 (owner: 10RobH) [21:49:06] binasher: But users are (or should be) aware that user databases on replicas are transient [21:49:22] We do tell them to back up their databases. [21:49:30] Coren: ah, cool [21:49:56] won't worry then :) the whole thing just makes me think of community wrath [21:50:01] which can be pretty damned wrathful [21:50:22] we may be able to resurrect some users dbs. after repl is back up and innodb consistent, can investigate. hence keeping .bad [21:51:23] cool [21:51:34] RobH: really, wikimediastories?! what happened to my shopping list? :-) [21:52:35] Warning: include_once(): Failed opening '/a/common/php-1.22wmf17/extensions/Flow/Flow.php' for inclusion (include_path='/a/common/php-1.22wmf17/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.22wmf17:/usr/local/lib/php:/usr/share/php') in /a/common/php-1.22wmf17/maintenance/mergeMessageFileList.php on line 110 [21:52:35] Unable to read /a/common/php-1.22wmf17/extensions/Flow/Flow.php [21:52:35] Update of MediaWiki localisation messages failed [21:52:37] That's a no [21:54:05] chrismcmahon: The "simplest" option at this point is to branch and include it in wmf branches, then add to extension-list [21:54:15] Which really isn't sensible/maintainable going forward [21:54:23] especially with other extensions under development [21:54:42] (03PS1) 10Akosiaris: Disable HTTPS on etherpad.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/84873 [21:55:19] Reedy: hrm. it's sort of usable for current purposes but I think it is a great use case for beta, to host unreleased extensions there. [21:55:26] (03PS2) 10Cmjohnson: nickel: add self ('olivneh') w/sudo [operations/puppet] - 10https://gerrit.wikimedia.org/r/83201 (owner: 10Ori.livneh) [21:55:27] exactly [21:55:47] (03CR) 10Cmjohnson: [C: 032] nickel: add self ('olivneh') w/sudo [operations/puppet] - 10https://gerrit.wikimedia.org/r/83201 (owner: 10Ori.livneh) [21:55:48] if if list-file didn't take a filename [21:55:52] * chrismcmahon waves hands vaguely but enthusiastically [21:56:13] we could do something like extensions=`cat extension-list*` [21:56:46] (03PS2) 10BBlack: Make vhtcpd collector and pyconf agree on titles. [operations/puppet] - 10https://gerrit.wikimedia.org/r/83932 (owner: 10BryanDavis) [21:56:53] (03CR) 10BBlack: [C: 032] Make vhtcpd collector and pyconf agree on titles. [operations/puppet] - 10https://gerrit.wikimedia.org/r/83932 (owner: 10BryanDavis) [21:57:02] (03PS1) 10RobH: RT#4942 wikimediastories.com/net/org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84874 [21:57:18] bblack: thanks [21:57:33] Or we add a new constant type of thing [21:57:52] bd808: I didn't notice it until just now for some reason [21:57:54] $MW_COMMON_EXTENSION_LIST and that can vary between production and not [21:58:03] Maintain a copy of extension-list for labs usage [21:58:05] bd808: but still trying to figure out why it hasn't rebase + jenkins verify yet [21:58:15] (03PS2) 10RobH: RT#4942 wikimediastories.com/net/org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84874 [21:58:16] Reedy: stands to reason that since we have e.g. CommonSettings-labs.php and InitialiseSettings-labs.php that we could create a similar mechanism for extension-list-labs also [21:58:17] (03PS3) 10BBlack: Make vhtcpd collector and pyconf agree on titles. [operations/puppet] - 10https://gerrit.wikimedia.org/r/83932 (owner: 10BryanDavis) [21:58:33] bblack: I forgot to add you as a reviewer until today [21:58:38] ah ok [21:58:42] chrismcmahon: Unfortunately we're not in mediawiki land at this point [21:58:53] so much for reason [21:59:44] chrismcmahon: A global maybe? [21:59:46] (03CR) 10BBlack: [C: 032] Make vhtcpd collector and pyconf agree on titles. [operations/puppet] - 10https://gerrit.wikimedia.org/r/83932 (owner: 10BryanDavis) [21:59:58] when we get into the maintenance script we are in mediawiki land [22:00:17] There's no real reason that we have to pass in extension-list as a command line parameter [22:00:34] cmjohnson1: woot, thanks. [22:00:41] ! [22:01:32] $wgExtensionList = getRealmSpecificFilename( $wmfConfigDir ."/extension-list" ); [22:01:36] yw...lmk if you have an problems [22:02:40] chrismcmahon: let me code it up [22:02:57] Need to think of a name for the global [22:03:51] (03PS1) 10Lcarr: correcting netboot gateway [operations/puppet] - 10https://gerrit.wikimedia.org/r/84876 [22:04:31] (03CR) 10Lcarr: [C: 032] correcting netboot gateway [operations/puppet] - 10https://gerrit.wikimedia.org/r/84876 (owner: 10Lcarr) [22:04:59] (03CR) 10Lcarr: [V: 032] correcting netboot gateway [operations/puppet] - 10https://gerrit.wikimedia.org/r/84876 (owner: 10Lcarr) [22:06:43] grrrrrrrrrrrrrrit [22:08:19] (03PS2) 10Akosiaris: Disable HTTPS on etherpad.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/84873 [22:08:20] (03PS1) 10Akosiaris: Reduce retry timeout for etherpad frontend apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/84882 [22:10:02] dirsyncing Zero [22:10:47] !log yurik synchronized php-1.22wmf16/extensions/ZeroRatedMobileAccess/ [22:10:51] Logged the message, Master [22:13:19] !log yurik synchronized php-1.22wmf17/extensions/ZeroRatedMobileAccess/ [22:13:23] Logged the message, Master [22:15:36] gr zero deployed. [22:15:40] greg-g, ^& [22:15:59] * yuri_k looks for a new keyboard that knows what i want to type instead of typing what i press [22:17:06] :) [22:19:55] (03Abandoned) 10Akosiaris: Puppetize Bugzilla [operations/puppet] - 10https://gerrit.wikimedia.org/r/62404 (owner: 10Ori.livneh) [22:45:34] (03CR) 10RobH: [C: 032] RT#4942 wikimediastories.com/net/org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84874 (owner: 10RobH) [22:52:01] (03PS1) 10RobH: RT#4942 wikimediastories redirection [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84891 [22:52:53] (03PS2) 10RobH: RT#4942 wikimediastories redirection [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84891 [22:52:54] typosssss [22:53:32] (03CR) 10RobH: [C: 032] RT#4942 wikimediastories redirection [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84891 (owner: 10RobH) [22:53:43] sanitarium is always a name that worries me [22:58:02] (03PS1) 10RobH: Revert "RT#4942 wikimediastories redirection" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84894 [22:58:11] (03Abandoned) 10RobH: Revert "RT#4942 wikimediastories redirection" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84894 (owner: 10RobH) [22:58:17] bleh, didnt mean to hit revert. [23:00:08] (03PS1) 10RobH: reverting my wikimediastories multiple patchsets and commits [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84895 [23:01:05] LeslieCarr: no clue its black screen for me, i think its still in tftp load [23:01:18] oh, wrong channel [23:02:24] hehe [23:02:51] trying again [23:04:37] (03PS1) 10Reedy: Use $wgMessageFileList [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84898 [23:05:34] (03CR) 10RobH: [C: 032] "thats what i get for self reviewing (having to undo my work)" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84895 (owner: 10RobH) [23:05:57] mutante: so no dice on that attempt [23:06:08] i just reverted all my changes on fenari so any other apaches dont get bad config [23:06:11] i think this is further than its gone before - Sep 18 23:04:53 carbon atftpd[8025]: Serving precise-installer/ubuntu-installer/amd64/linux to 198.35.26.5:49154 [23:06:14] its livehacked on mw1229 to try it out [23:06:22] (03PS1) 10Reedy: Remove usage of list-file in mergeMessageFileList.php [operations/puppet] - 10https://gerrit.wikimedia.org/r/84900 [23:06:32] LeslieCarr: is that now? cuz tftpd stuff isnt need [23:06:33] new [23:06:39] it completes all tftp for the pxe boot [23:06:45] the network error is installer based [23:06:54] (ie: shouldnt have anything to do with tftpd) [23:07:11] it takes forever to complete over the link, but it does complete [23:07:22] Lightning deploy time! [23:07:41] oh it did get that far before [23:07:42] nm [23:07:46] I'm the only one on the list for today so I'm just gonna go ahead [23:07:47] i thought it got to a step earlier [23:07:57] well at least it's still tftp'ing …. sloooooooowly [23:08:07] yea it finishes dealing with carbon [23:09:36] (03PS1) 10Dzahn: remove SSLCACertificatePath from Apache config templates. [operations/puppet] - 10https://gerrit.wikimedia.org/r/84901 [23:09:51] so dhcpd config and the autoinstaller file in files/autoinstall/subnets/public1-ulsfo match now [23:09:54] !log catrope synchronized php-1.22wmf17/extensions/VisualEditor/ 'Deploy cherry-picked bug fixes' [23:09:58] Logged the message, Master [23:09:58] it should work. [23:10:25] All done [23:15:32] (03CR) 10Akosiaris: [C: 032] remove SSLCACertificatePath from Apache config templates. [operations/puppet] - 10https://gerrit.wikimedia.org/r/84901 (owner: 10Dzahn) [23:16:31] RobH: [23:19:48] 9 mins for jenkins ??? [23:20:16] isn't 9 minutes for a jenkins-bot verified a bit too much ? [23:21:03] which one? the one above looks like 3min [23:21:41] (03PS1) 10RobH: RT#4942 wikimediastories [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84902 [23:22:28] well 2 mins for all jenkins jobs. but on the whole you submitted at 4:09 PM and jenkins-bot verified ad 4:19 PM [23:22:33] (03CR) 10Dzahn: [C: 031] "testing 2 urls on 1 servers, totalling 2 requests" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84902 (owner: 10RobH) [23:23:12] (03CR) 10RobH: [C: 032] RT#4942 wikimediastories [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84902 (owner: 10RobH) [23:23:22] 4:18PM ... typo sorry [23:23:41] akosiaris1: ah true, yea [23:24:59] (03PS1) 10RobH: Revert "RT#4942 wikimediastories" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84903 [23:25:25] (03CR) 10RobH: [C: 032] "1 step forward, then fall backwards down the hill" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84903 (owner: 10RobH) [23:44:56] !log Jenkins applied labels "hasPhpcs" and "hasPhpUnit" to gallium slave [23:45:00] Logged the message, Master [23:47:09] RECOVERY - mysqld processes on labsdb1003 is OK: PROCS OK: 3 processes with command name mysqld [23:47:33] paravoid: Do you have time tomorrow to give me that chalk talk on the image scaler setup that we talked about? [23:51:43] !log Jenkins: migrated mediawiki extension 'lint' jobs to fetch Zuul reference over git:// protocol. Migrating jslint ones. [23:51:46] Logged the message, Master [23:57:16] paravoid: not sure why there are so many timeline tmp files