[00:01:44] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 6.448 second response time [00:05:08] (03PS2) 10BryanDavis: Make elasticsearch ganglia monitor compatible with logstash [operations/puppet] - 10https://gerrit.wikimedia.org/r/113471 [00:05:27] (03CR) 10Ori.livneh: [C: 032 V: 032] Make elasticsearch ganglia monitor compatible with logstash [operations/puppet] - 10https://gerrit.wikimedia.org/r/113471 (owner: 10BryanDavis) [00:08:37] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 212480 bytes in 6.978 second response time [00:12:38] (03CR) 10Ori.livneh: "Could we go a step further and discard the comment field? We haven't had comments in the file for a while; it is a throwback to a time whe" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 (owner: 10Chad) [00:20:53] (03PS2) 10Chad: WIP: Remove unused third parameter for wikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 [00:25:05] (03CR) 10Ori.livneh: [C: 04-1] WIP: Remove unused third parameter for wikiversions (033 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 (owner: 10Chad) [00:27:21] (03PS3) 10Chad: Remove unused third/fourth parameters for wikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 [00:29:10] (03PS4) 10Chad: Remove unused third/fourth parameters for wikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 [00:36:15] (03PS5) 10Ori.livneh: Remove unused third/fourth parameters for wikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 (owner: 10Chad) [00:37:34] (03PS6) 10Chad: Remove unused third/fourth parameters for wikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 [00:41:12] (03CR) 10Reedy: "We could have just reverted https://github.com/wikimedia/operations-mediawiki-multiversion/commit/1eeaff2e920cc129857f06fdddb8ca4b1ec9ef9e" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 (owner: 10Chad) [00:44:11] (03CR) 10Ori.livneh: [V: 032] Remove unused third/fourth parameters for wikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 (owner: 10Chad) [00:44:49] (03CR) 10Ori.livneh: [C: 032] Remove unused third/fourth parameters for wikiversions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112955 (owner: 10Chad) [00:44:57] !log ori updated /a/common to {{Gerrit|Id87a90474}}: Remove unused third/fourth parameters for wikiversions [00:45:06] Logged the message, Master [00:54:06] (03PS1) 10Ori.livneh: Handle empty lines gracefully [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113497 [00:54:29] Reedy: ^ [00:55:09] it just restores a piece of what the previous patch removed [00:55:40] (03CR) 10Reedy: [C: 032] Handle empty lines gracefully [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113497 (owner: 10Ori.livneh) [00:55:47] (03Merged) 10jenkins-bot: Handle empty lines gracefully [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113497 (owner: 10Ori.livneh) [00:55:59] !log ori updated /a/common to {{Gerrit|I7e014f29e}}: Handle empty lines gracefully [00:56:06] Logged the message, Master [00:57:06] verified by rebuilding wikiversions.cdb and comparing md5sums [00:57:43] 3200 characters saved! [00:58:48] well, wikiversions.dat -> wikiversions.json next [00:58:56] :D [00:58:58] also: that's... not over 9,000 [01:02:24] mark: hmm, with Swift, I see sporadic 503s on container HEADs and 404s on PUTs (usually some backend error or sqlite lock timeout). Seems like that only happens when one of the backends is flapping around or something. [01:03:52] * AaronSchulz also needs to track down the cause of those random 401s [01:06:36] * AaronSchulz sees nothing interesting in nagios [01:10:39] (03PS1) 10Jean-Frédéric: Add Musées de la Haute-Saône to wgCopyUploadsDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113500 [01:20:47] ori: the resolution of https://bugs.php.net/bug.php?id=54626 is funny [01:23:35] * AaronSchulz tries to make sense of https://bugzilla.wikimedia.org/show_bug.cgi?id=60988 [01:24:04] Weird shit, yo [01:28:01] Reedy: No need to talk about AaronSchulz like that >.> [01:28:23] most of those bugs looks like APC corruption [01:28:38] Most of them can probably just be closed [01:28:45] https://bugzilla.wikimedia.org/show_bug.cgi?id=60997 [01:28:55] please close any that you think should be then [01:29:08] I don't think any of them have re-occurred [01:29:22] So the 5 open can probably go plus the tracking bug [01:29:25] you can always do apache-graceful is this stuff happens [01:29:41] yep, I fixed the actual slowness bug [01:30:06] Ooh, duplicates field on bugzilla [01:33:44] (03PS13) 10BBlack: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik) [01:46:37] PROBLEM - Puppet freshness on dysprosium is CRITICAL: Last successful Puppet run was Fri 14 Feb 2014 07:45:00 PM UTC [01:47:23] (03PS14) 10BBlack: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik) [02:28:23] (03PS15) 10BBlack: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik) [02:29:05] !log LocalisationUpdate completed (1.23wmf13) at 2014-02-15 02:29:05+00:00 [02:29:17] Logged the message, Master [02:39:10] (03PS16) 10BBlack: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik) [02:50:08] !log LocalisationUpdate completed (1.23wmf14) at 2014-02-15 02:50:08+00:00 [02:50:16] Logged the message, Master [03:31:58] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-02-15 03:31:58+00:00 [03:32:06] Logged the message, Master [03:38:53] (03PS17) 10BBlack: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik) [03:40:29] (03PS2) 10Yurik: Add X-CS response for all requests that came from Zero network [operations/puppet] - 10https://gerrit.wikimedia.org/r/112805 [03:40:43] (03CR) 10BBlack: [C: 032 V: 032] Add X-CS response for all requests that came from Zero network [operations/puppet] - 10https://gerrit.wikimedia.org/r/112805 (owner: 10Yurik) [03:41:15] (03PS18) 10Yurik: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 [04:06:22] (03PS19) 10BBlack: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik) [04:07:47] (03CR) 10BBlack: [C: 032 V: 032] Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 (owner: 10Yurik) [04:30:27] PROBLEM - RAID on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:31:17] RECOVERY - RAID on db1021 is OK: OK: optimal, 1 logical, 2 physical [04:47:37] PROBLEM - Puppet freshness on dysprosium is CRITICAL: Last successful Puppet run was Fri 14 Feb 2014 07:45:00 PM UTC [05:09:28] (03PS1) 10Springle: s1 assign db1034 [operations/puppet] - 10https://gerrit.wikimedia.org/r/113522 [05:10:48] (03CR) 10Springle: [C: 032] s1 assign db1034 [operations/puppet] - 10https://gerrit.wikimedia.org/r/113522 (owner: 10Springle) [05:14:15] !log xtrabackup clone db1055 to db1010 [05:14:25] Logged the message, Master [07:48:11] PROBLEM - Puppet freshness on dysprosium is CRITICAL: Last successful Puppet run was Fri 14 Feb 2014 07:45:00 PM UTC [10:18:09] (03CR) 10Odder: [C: 031] "\o/" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113500 (owner: 10Jean-Frédéric) [10:49:10] PROBLEM - Puppet freshness on dysprosium is CRITICAL: Last successful Puppet run was Fri 14 Feb 2014 07:45:00 PM UTC [12:14:30] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [12:19:45] (03Restored) 10Siebrand: Enable EducationProgram on Dutch language Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71605 (owner: 10Siebrand) [12:21:39] (03PS2) 10Siebrand: Enable EducationProgram on Dutch language Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71605 [12:24:12] (03PS1) 10Siebrand: Ignore PhpStorm files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113531 [12:51:17] mutante: Late but hey, great job on the Bugzilla upgrade! :) [12:58:56] (03CR) 10Siebrand: "There appears to be no objection." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71605 (owner: 10Siebrand) [13:12:31] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [13:50:10] PROBLEM - Puppet freshness on dysprosium is CRITICAL: Last successful Puppet run was Fri 14 Feb 2014 07:45:00 PM UTC [14:47:59] (03PS1) 10Springle: s1 substitute db1034 for db1055 during schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113537 [14:48:26] (03CR) 10Springle: [C: 032] s1 substitute db1034 for db1055 during schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113537 (owner: 10Springle) [14:48:32] (03Merged) 10jenkins-bot: s1 substitute db1034 for db1055 during schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113537 (owner: 10Springle) [14:49:59] !log springle synchronized wmf-config/db-eqiad.php 's1 substitute db1034 for db1055 during schema changes' [14:50:08] Logged the message, Master [15:03:00] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [15:04:40] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 36.39 ms [15:06:50] PROBLEM - Apache HTTP on mw27 is CRITICAL: Connection refused [15:07:50] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.439 second response time [16:51:10] PROBLEM - Puppet freshness on dysprosium is CRITICAL: Last successful Puppet run was Fri 14 Feb 2014 07:45:00 PM UTC [16:57:45] (03PS1) 10Odder: Create an ArbCom group on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113542 [17:23:53] what [18:47:43] (03PS3) 10Ryan Lane: Code documentation for trebuchet's deployment module [operations/puppet] - 10https://gerrit.wikimedia.org/r/112855 [19:22:04] (03CR) 10Ori.livneh: [C: 032] Code documentation for trebuchet's deployment module [operations/puppet] - 10https://gerrit.wikimedia.org/r/112855 (owner: 10Ryan Lane) [19:26:46] (03PS1) 10Yurik: Zero: Add TEST provider as supporting SSL [operations/puppet] - 10https://gerrit.wikimedia.org/r/113547 [19:29:41] (03PS2) 10Yurik: Zero: Add TEST provider as supporting SSL [operations/puppet] - 10https://gerrit.wikimedia.org/r/113547 [19:29:48] (03CR) 10BBlack: [C: 032 V: 032] Zero: Add TEST provider as supporting SSL [operations/puppet] - 10https://gerrit.wikimedia.org/r/113547 (owner: 10Yurik) [19:31:25] ori: can I merge the trebuchet thing? or you can push mine if you're already in there [19:43:10] I'm gonna take silence for a yes :P [19:52:10] PROBLEM - Puppet freshness on dysprosium is CRITICAL: Last successful Puppet run was Fri 14 Feb 2014 07:45:00 PM UTC [20:11:22] ori, Gloria: Wanted to have a go at https://bugzilla.wikimedia.org/show_bug.cgi?id=18831, would it be possible to do that in the Bugzilla labs project? [20:54:27] (03CR) 10Nemo bis: [C: 031] Create an ArbCom group on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113542 (owner: 10Odder) [21:05:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:07:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:09:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:11:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:13:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:15:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:17:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:19:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:21:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:22:01] (03PS1) 10Matanya: remove shell account for lwelling and access to stat1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/113627 [21:22:58] (03PS2) 10Matanya: remove shell account for lwelling and access to stat1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/113627 [21:23:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:25:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:26:57] apergos: around? [21:27:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 08:58:33 PM UTC [21:28:35] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Sat Feb 15 21:28:32 UTC 2014 [21:30:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 09:28:32 PM UTC [21:32:04] PROBLEM - Puppet freshness on db38 is CRITICAL: Last successful Puppet run was Sat 15 Feb 2014 09:28:32 PM UTC [21:46:33] (03PS1) 10Matanya: remove shell access and key for mgrover [operations/puppet] - 10https://gerrit.wikimedia.org/r/113636 [21:47:36] (03PS3) 10Matanya: remove shell account for lwelling and access to stat1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/113627 [21:52:46] (03PS1) 10Matanya: removed orion shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113637 [21:54:01] (03PS2) 10Matanya: removed orion shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113637 [21:57:02] (03PS1) 10Matanya: remove smerritt shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113638 [21:58:12] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Sat Feb 15 21:58:03 UTC 2014 [22:02:11] (03PS1) 10Matanya: remove darrell shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113639 [22:03:07] So many patches for one ticket [22:04:13] matanya: we might run out of numbers if you carry on at this rate [22:04:45] Nemo_bis: it is seperated so some can be merged while some are rejected [22:04:59] Yeah, please consider the cost of hundreds devs having to write 7 digits instead of 6 [22:05:19] Sure, I was just wonder how much blame I'll havae for filing that ticket :P [22:05:34] Reedy: i think the wmf can count on you only :P [22:05:46] Also, doesn't RT: work in footer? Is there a bug for that? [22:06:11] no Nemo_bis only RT #number works [22:06:18] (03CR) 10Reedy: [C: 04-1] removed orion shell account (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/113637 (owner: 10Matanya) [22:06:57] (03CR) 10Reedy: [C: 04-1] remove shell access and key for mgrover (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/113636 (owner: 10Matanya) [22:07:17] (03CR) 10Reedy: [C: 04-1] remove shell account for lwelling and access to stat1 (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/113627 (owner: 10Matanya) [22:07:53] (03CR) 10Reedy: [C: 04-1] remove darrell shell account (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/113639 (owner: 10Matanya) [22:08:22] (03CR) 10Reedy: [C: 04-1] remove smerritt shell account (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/113638 (owner: 10Matanya) [22:09:45] (03PS4) 10Matanya: remove shell account for lwelling and access to stat1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/113627 [22:10:57] (03PS2) 10Matanya: remove shell access and key for mgrover [operations/puppet] - 10https://gerrit.wikimedia.org/r/113636 [22:10:58] tabs are evil [22:12:21] (03PS3) 10Matanya: removed orion shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113637 [22:13:08] (03PS2) 10Matanya: remove smerritt shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113638 [22:13:50] (03PS2) 10Matanya: remove darrell shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/113639 [22:14:02] thank you Reedy [22:14:08] heh [22:20:04] bblack: fine to merge (and i see that you did). sorry bout that. [22:20:21] night all [22:27:12] !log reedy updated /a/common to {{Gerrit|I04d387adf}}: s1 substitute db1034 for db1055 during schema changes [22:27:20] Logged the message, Master [22:27:56] (03PS1) 10Reedy: Remove 1.23wmf1 through 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113640 [22:28:05] (03CR) 10Reedy: [C: 032] Remove 1.23wmf1 through 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113640 (owner: 10Reedy) [22:28:12] (03Merged) 10jenkins-bot: Remove 1.23wmf1 through 1.23wmf5 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113640 (owner: 10Reedy) [22:30:20] !log reedy updated /a/common to {{Gerrit|Id33b8287c}}: Remove 1.23wmf1 through 1.23wmf5 [22:30:23] (03PS1) 10Reedy: Remove old 1.22wmf22 and 1.22wmf5 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113641 [22:30:28] Logged the message, Master [22:30:36] (03CR) 10Reedy: [C: 032] Remove old 1.22wmf22 and 1.22wmf5 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113641 (owner: 10Reedy) [22:30:43] (03Merged) 10jenkins-bot: Remove old 1.22wmf22 and 1.22wmf5 symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113641 (owner: 10Reedy) [22:38:42] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 957.413585188 [22:52:12] PROBLEM - Puppet freshness on dysprosium is CRITICAL: Last successful Puppet run was Fri 14 Feb 2014 07:45:00 PM UTC [22:59:42] Krenair: What's your proposed implementation approach? [23:00:06] Krenair: I think investigating or building a Bugzilla plugin would be best. [23:00:26] Krenair: Parsing Bugzilla mail is certainly not a great approach (the current approach). [23:01:33] Well I'm certainly not rewriting wikibugs, so changing away from parsing mail is not happening [23:02:00] I was going to look into hacking in support for sending the name in the email to BZ. probably in a header. [23:03:01] Email *to* BZ? [23:03:21] hacking in support [for sending the name in the email] to BZ [23:03:52] ok [23:03:58] Maybe X-Bugzilla-Who is even configurable [23:08:36] Not in any way redhat and mozilla use though [23:14:20] (03CR) 10Ori.livneh: "When do you expect the conversation with Facebook to yield a usable package?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112314 (owner: 10Ori.livneh) [23:34:29] Krenair: Erggg, I really wish someone would. [23:36:03] Gloria, okay great. I'm interested in at least looking into doing that, but I don't have BZ or wikibugs installed [23:36:32] Well, there's the boogs.wmflabs.org installation still, maybe. [23:36:38] No idea how to get access. [23:36:47] Installing wikibugs is mostly a matter of having Perl, I think. [23:36:52] Kernel exploits