[00:00:02] (03CR) 10Dzahn: "not yet per talk with chris" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148289 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [00:00:17] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:03:32] (03CR) 10Mwalker: [C: 032] Put quotes around strings in YAML that start with % [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/150081 (owner: 10Yuvipanda) [00:03:34] (03Merged) 10jenkins-bot: Put quotes around strings in YAML that start with % [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/150081 (owner: 10Yuvipanda) [00:05:53] (03PS1) 10Ori.livneh: mediawiki: correct name of configuration key in hhvm.ini [operations/puppet] - 10https://gerrit.wikimedia.org/r/150092 [00:06:13] (03CR) 10Ori.livneh: [C: 032 V: 032] mediawiki: correct name of configuration key in hhvm.ini [operations/puppet] - 10https://gerrit.wikimedia.org/r/150092 (owner: 10Ori.livneh) [00:06:27] (03CR) 10Tim Landscheidt: [C: 04-1] "Ori has some points, but I need to sleep first :-)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120186 (owner: 10Tim Landscheidt) [00:07:03] (03PS2) 10Dzahn: wikidata.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150071 [00:08:24] greg-g, yuvi has a patch for jouncebot that implements a 'last' command -- but it doesn't actually do anything; what were you intending? [00:08:58] (03PS3) 10Dzahn: wikidata.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150071 [00:11:38] mwalker: it was a joke :) [00:11:53] bd808 was looking for what would be better termed "previous" [00:12:05] (03PS2) 10Dzahn: wikivoyage-old.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150072 [00:12:26] (03CR) 10BBlack: [C: 032] "confirmed whitespace-only" [operations/dns] - 10https://gerrit.wikimedia.org/r/150071 (owner: 10Dzahn) [00:12:33] ah; yes [00:12:40] that needs to happen [00:12:44] *before I depart [00:12:55] because without a deadline [00:12:56] * greg-g shrugs [00:12:57] it'll take a long time [00:12:58] yeah :) [00:14:03] (03CR) 10BBlack: [C: 032] "Confirmed whitespace-only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150072 (owner: 10Dzahn) [00:16:51] (03CR) 10BBlack: [C: 031] wikimedia.ee - own zonefile and set external MX [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 (owner: 10Dzahn) [00:16:56] (03PS1) 10Dzahn: wikivoyage-old.org - fix alignment [operations/dns] - 10https://gerrit.wikimedia.org/r/150093 [00:17:44] (03CR) 10Dzahn: "JeremyB: no, the source i had copied from still had the tabs when i made it and i removed those on purpose, you re-added them. but meanwhi" [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 (owner: 10Dzahn) [00:18:52] (03CR) 10Dzahn: [C: 032] wikivoyage-old.org - fix alignment [operations/dns] - 10https://gerrit.wikimedia.org/r/150093 (owner: 10Dzahn) [00:20:49] (03PS7) 10Dzahn: wikimedia.ee - own zonefile and set external MX [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 [00:20:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [00:22:22] (03PS1) 10Dzahn: wikimedia.com - retab and alignment [operations/dns] - 10https://gerrit.wikimedia.org/r/150094 [00:23:38] (03PS2) 10Dzahn: wikimedia.com - retab and alignment [operations/dns] - 10https://gerrit.wikimedia.org/r/150094 [00:25:36] (03PS1) 10Dzahn: wmnet - retab only [operations/dns] - 10https://gerrit.wikimedia.org/r/150095 (https://bugzilla.wikimedia.org/68769) [00:27:08] (03PS1) 10Dzahn: wmftest.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150096 [00:27:42] !log aaron Synchronized php-1.24wmf15/includes: f754c239ce93fc5f2db19e93f4fe8a1d1ba7bc27 (duration: 00m 06s) [00:27:48] Logged the message, Master [00:27:54] !log aaron Synchronized php-1.24wmf15/maintenance: f754c239ce93fc5f2db19e93f4fe8a1d1ba7bc27 (duration: 00m 04s) [00:27:59] Logged the message, Master [00:30:17] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 1 failures [00:30:19] (03PS1) 10Dzahn: wikivoyage.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150099 [00:33:36] (03PS1) 10Dzahn: wiktionary.org - retab and align [operations/dns] - 10https://gerrit.wikimedia.org/r/150101 [00:33:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [00:35:57] (03PS1) 10Dzahn: wikiversity.org - retab and align [operations/dns] - 10https://gerrit.wikimedia.org/r/150103 [00:36:33] (03CR) 10Scottlee: [C: 031] wikimedia.com - retab and alignment [operations/dns] - 10https://gerrit.wikimedia.org/r/150094 (owner: 10Dzahn) [00:37:04] (03PS1) 10Dzahn: wikisource.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150105 [00:38:03] (03CR) 10Dzahn: [C: 032] wikimedia.com - retab and alignment [operations/dns] - 10https://gerrit.wikimedia.org/r/150094 (owner: 10Dzahn) [00:41:00] (03PS1) 10Dzahn: 10.in-addr.arpa - retab only [operations/dns] - 10https://gerrit.wikimedia.org/r/150107 [00:41:33] (03PS1) 10Dzahn: 154.80.208.in-addr.arpa - retab only [operations/dns] - 10https://gerrit.wikimedia.org/r/150108 [00:44:39] (03PS1) 10Dzahn: 174.198.91.in-addr.arpa - retab only [operations/dns] - 10https://gerrit.wikimedia.org/r/150109 [00:44:41] !log aaron Synchronized php-1.24wmf15/maintenance/runJobs.php: fcfa3153e53dc70e6cd190a087e7bd577fe380fb (duration: 00m 03s) [00:44:46] Logged the message, Master [00:48:17] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [00:48:52] (03CR) 10Dzahn: [C: 032] Fix str_replace call [operations/apache-config] - 10https://gerrit.wikimedia.org/r/147491 (owner: 10Reedy) [00:55:10] ori: https://gerrit.wikimedia.org/r/#/c/148741/ ? [00:55:17] can all stuff there be abandoned? [00:55:38] because the repo is discontinued.. or should it be converted to puppet repo [00:55:43] a change in the other repo, i mean [00:56:04] basically all these https://gerrit.wikimedia.org/r/#/q/project:operations/apache-config+status:open,n,z [00:56:33] mutante: beta hasn't been ported yet [00:56:41] notice the branch on that particular change is 'betacluster' [00:56:57] but that particular patch can be abandoned for another reason -- namely, it doesn't work very well :) [00:57:10] I think there's only Bryans patch that wants saving from the rest [00:57:22] ah, i see @ beta [00:57:57] ori: gotcha:) [00:58:02] Reedy: haha, what about us-ne though [00:58:10] or should i say.. ne-us :) [00:58:28] KILLKILLKILL [00:58:46] (03CR) 10Dzahn: [C: 04-1] "apache-config repo has been abandoned (see new file README_BEFORE_EDITING), please abandon or convert to a change in operations/puppet" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/109460 (https://bugzilla.wikimedia.org/60222) (owner: 10Tim Landscheidt) [00:58:58] (03CR) 10Dzahn: [C: 04-1] "apache-config repo has been abandoned (see new file README_BEFORE_EDITING), please abandon or convert to a change in operations/puppet" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/108880 (https://bugzilla.wikimedia.org/43266) (owner: 10Tim Landscheidt) [00:59:13] (03CR) 10Dzahn: [C: 04-1] "apache-config repo has been abandoned (see new file README_BEFORE_EDITING), please abandon or convert to a change in operations/puppet" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/108465 (https://bugzilla.wikimedia.org/60238) (owner: 10Tim Landscheidt) [00:59:32] (03CR) 10Dzahn: [C: 04-1] "apache-config repo has been abandoned (see new file README_BEFORE_EDITING), please abandon or convert to a change in operations/puppet" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/111925 (owner: 10BryanDavis) [00:59:44] (03CR) 10Dzahn: [C: 04-1] "apache-config repo has been abandoned (see new file README_BEFORE_EDITING), please abandon or convert to a change in operations/puppet" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133991 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [00:59:52] (03CR) 10Dzahn: [C: 04-1] "apache-config repo has been abandoned (see new file README_BEFORE_EDITING), please abandon or convert to a change in operations/puppet" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133981 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [01:00:04] (03CR) 10Dzahn: [C: 04-1] "apache-config repo has been abandoned (see new file README_BEFORE_EDITING), please abandon or convert to a change in operations/puppet" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/24407 (owner: 10Jeremyb) [01:00:30] Ubuntu 14.04.1 LTS [01:00:34] Didn't realise we were there already [01:03:06] Reedy: yea, we already have f.e. phabricator running on trusty [01:04:16] Reedy: https://gerrit.wikimedia.org/r/#/c/133992/ ? :p [01:04:45] I'd kill it too [01:04:54] Seeing as they don't know what they want [01:05:02] but what are they going to do now? [01:05:02] That, and surely it won't merge cleanly now anyway? ;) [01:05:09] not let us host it all ? [01:05:16] too much drama? sigh [01:05:30] yeah [01:05:30] lol [01:05:34] thinks it's rude to abandon other people's changes [01:05:38] but.. yea [01:05:51] Antoine goes through and abandons some of mine from time to time :) [01:05:58] arg, but all i said was "make it consistent with other US chapter" [01:06:04] who would have known [01:06:40] LOL, i opened the bug and read 1 random sentence and it was [01:06:42] "re MZMcBride's comments about having chapter wikis at all:" [01:08:47] (03CR) 10Dzahn: [C: 04-2] "the bug says "Please place a hold on this"" [operations/dns] - 10https://gerrit.wikimedia.org/r/133992 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [01:11:38] (03CR) 10Dzahn: [C: 04-2] Apache set up for us-ne [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133981 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [01:11:50] (03CR) 10Dzahn: [C: 04-2] Redirect usne to us-ne [operations/apache-config] - 10https://gerrit.wikimedia.org/r/133991 (https://bugzilla.wikimedia.org/64557) (owner: 10John F. Lewis) [01:13:20] (03CR) 10Dzahn: [C: 04-1] "please convert to admins.yaml:)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122621 (owner: 10Reedy) [01:14:06] Hahaha [01:15:59] (03CR) 10Dzahn: [C: 031] "https://legalteam.wikimedia.org/w/touch.php" [operations/puppet] - 10https://gerrit.wikimedia.org/r/147488 (owner: 10Reedy) [01:18:54] (03CR) 10Dzahn: Add robots.txt rewrite rule where wiki is public (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/147487 (owner: 10Reedy) [01:22:06] (03CR) 10Dzahn: [C: 031] add index.html pages for various directories on dataset hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/144640 (owner: 10ArielGlenn) [01:23:42] (03CR) 10Dzahn: "we should run this on the puppet compile: https://integration.wikimedia.org/ci/view/operations/job/operations-puppet-catalog-compiler/ ->" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124001 (owner: 10Tim Landscheidt) [01:25:04] (03CR) 10Dzahn: [C: 031] "added _joe_" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148544 (owner: 10MaxSem) [01:27:03] (03CR) 10Dzahn: "what Alex said" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/145018 (owner: 10ArielGlenn) [01:36:07] PROBLEM - Disk space on ms-be3004 is CRITICAL: Timeout while attempting connection [01:36:47] PROBLEM - Host amssq39 is DOWN: PING CRITICAL - Packet loss = 100% [01:36:47] PROBLEM - Host amssq40 is DOWN: PING CRITICAL - Packet loss = 100% [01:36:47] PROBLEM - Host amssq37 is DOWN: PING CRITICAL - Packet loss = 100% [01:36:58] PROBLEM - Host amssq42 is DOWN: PING CRITICAL - Packet loss = 100% [01:36:58] PROBLEM - Host cp3017 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:08] PROBLEM - Host amssq43 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:08] PROBLEM - Host amssq31 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:08] PROBLEM - Host amssq41 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:08] PROBLEM - Host amssq44 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:08] PROBLEM - Host amssq32 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:15] (03PS8) 10BBlack: wikimedia.ee - own zonefile and set external MX [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 (owner: 10Dzahn) [01:37:17] PROBLEM - Host amssq33 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:18] PROBLEM - Host amssq45 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:18] PROBLEM - Host cp3013 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:24] ah.. what's that [01:37:27] PROBLEM - Host amssq46 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:27] PROBLEM - Host cp3014 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:27] PROBLEM - Host amssq34 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:27] PROBLEM - Host cp3015 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:27] PROBLEM - Host amssq47 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:27] (03CR) 10Scottlee: [C: 031] wikisource.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150105 (owner: 10Dzahn) [01:37:28] PROBLEM - Host amssq35 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:37] PROBLEM - Host amssq36 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:37] PROBLEM - Host cp3016 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:37] PROBLEM - Host cp3018 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:47] PROBLEM - Host amssq38 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:47] PROBLEM - Host ms-be3002 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:47] PROBLEM - Host ms-be3001 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:47] PROBLEM - Host ms-be3004 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:47] PROBLEM - Host lvs3003 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:48] PROBLEM - Host ms-fe3002 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:48] PROBLEM - Host lvs3001 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:49] PROBLEM - Host lvs3002 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:49] PROBLEM - Host lvs3004 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:50] PROBLEM - Host ms-be3003 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:50] PROBLEM - Host ms-fe3001 is DOWN: PING CRITICAL - Packet loss = 100% [01:37:53] (03CR) 10Scottlee: [C: 031] wikiversity.org - retab and align [operations/dns] - 10https://gerrit.wikimedia.org/r/150103 (owner: 10Dzahn) [01:38:48] (03CR) 10Scottlee: [C: 031] wiktionary.org - retab and align [operations/dns] - 10https://gerrit.wikimedia.org/r/150101 (owner: 10Dzahn) [01:39:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [500.0] [01:40:01] (03CR) 10Scottlee: [C: 031] 154.80.208.in-addr.arpa - retab only [operations/dns] - 10https://gerrit.wikimedia.org/r/150108 (owner: 10Dzahn) [01:41:58] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 16 failures [01:42:08] RECOVERY - Host amssq31 is UP: PING WARNING - Packet loss = 86%, RTA = 95.31 ms [01:42:08] RECOVERY - Host amssq43 is UP: PING WARNING - Packet loss = 86%, RTA = 97.23 ms [01:42:08] RECOVERY - Host amssq41 is UP: PING WARNING - Packet loss = 86%, RTA = 95.26 ms [01:42:08] RECOVERY - Host amssq44 is UP: PING WARNING - Packet loss = 86%, RTA = 95.42 ms [01:42:08] RECOVERY - Host amssq32 is UP: PING WARNING - Packet loss = 86%, RTA = 95.62 ms [01:42:08] RECOVERY - Host amssq37 is UP: PING WARNING - Packet loss = 80%, RTA = 95.42 ms [01:42:09] RECOVERY - Host amssq34 is UP: PING OK - Packet loss = 16%, RTA = 95.48 ms [01:42:09] RECOVERY - Host cp3017 is UP: PING OK - Packet loss = 16%, RTA = 95.96 ms [01:42:10] RECOVERY - Host cp3013 is UP: PING WARNING - Packet loss = 28%, RTA = 96.06 ms [01:42:10] RECOVERY - Host amssq33 is UP: PING WARNING - Packet loss = 28%, RTA = 97.00 ms [01:42:11] RECOVERY - Host amssq45 is UP: PING WARNING - Packet loss = 28%, RTA = 97.89 ms [01:42:17] RECOVERY - Host amssq46 is UP: PING OK - Packet loss = 0%, RTA = 96.38 ms [01:42:18] RECOVERY - Host cp3014 is UP: PING OK - Packet loss = 0%, RTA = 98.89 ms [01:42:18] RECOVERY - Host cp3015 is UP: PING OK - Packet loss = 0%, RTA = 95.39 ms [01:42:18] RECOVERY - Host amssq35 is UP: PING OK - Packet loss = 0%, RTA = 96.15 ms [01:42:18] RECOVERY - Host amssq47 is UP: PING OK - Packet loss = 0%, RTA = 95.48 ms [01:42:27] RECOVERY - Host amssq36 is UP: PING OK - Packet loss = 0%, RTA = 95.64 ms [01:42:27] RECOVERY - Host cp3016 is UP: PING OK - Packet loss = 0%, RTA = 95.78 ms [01:42:27] RECOVERY - Host ms-fe3001 is UP: PING OK - Packet loss = 0%, RTA = 95.85 ms [01:42:27] RECOVERY - Host amssq39 is UP: PING OK - Packet loss = 0%, RTA = 95.72 ms [01:42:27] RECOVERY - Host ms-be3004 is UP: PING OK - Packet loss = 0%, RTA = 95.86 ms [01:42:28] RECOVERY - Host amssq40 is UP: PING OK - Packet loss = 0%, RTA = 95.86 ms [01:42:28] RECOVERY - Host amssq38 is UP: PING OK - Packet loss = 0%, RTA = 96.07 ms [01:42:29] RECOVERY - Host cp3018 is UP: PING OK - Packet loss = 0%, RTA = 96.24 ms [01:42:29] RECOVERY - Host amssq42 is UP: PING OK - Packet loss = 0%, RTA = 96.09 ms [01:42:30] RECOVERY - Host ms-fe3002 is UP: PING OK - Packet loss = 0%, RTA = 96.26 ms [01:42:30] RECOVERY - Host ms-be3001 is UP: PING OK - Packet loss = 0%, RTA = 96.06 ms [01:42:31] RECOVERY - Host lvs3002 is UP: PING OK - Packet loss = 0%, RTA = 96.43 ms [01:42:31] RECOVERY - Host lvs3004 is UP: PING OK - Packet loss = 0%, RTA = 96.25 ms [01:42:32] RECOVERY - Host ms-be3002 is UP: PING OK - Packet loss = 0%, RTA = 96.31 ms [01:42:37] RECOVERY - Host ms-be3003 is UP: PING OK - Packet loss = 0%, RTA = 96.19 ms [01:42:37] RECOVERY - Host lvs3001 is UP: PING OK - Packet loss = 0%, RTA = 95.48 ms [01:42:37] RECOVERY - Host lvs3003 is UP: PING OK - Packet loss = 0%, RTA = 95.48 ms [01:43:08] that's cool.. but [01:43:30] all the addrs that went down (as opposed to those that didn't) were on the private1-esams subnet [01:43:42] the public access and the ones still on the older public subnet didn't down in icinga [01:44:13] ah.. eh [01:44:58] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 28 failures [01:44:58] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 18 failures [01:44:59] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 14 failures [01:44:59] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 32 failures [01:44:59] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 27 failures [01:44:59] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 31 failures [01:45:27] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 21 failures [01:45:27] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 7 failures [01:45:27] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 27 failures [01:45:27] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 29 failures [01:45:27] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 35 failures [01:45:58] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 6 failures [01:45:58] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 3 failures [01:45:58] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 1 failures [01:45:58] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 14 failures [01:46:11] lol [01:46:15] gj puppet [01:46:17] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 6 failures [01:46:18] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 1 failures [01:46:27] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 13 failures [01:46:27] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 2 failures [01:46:58] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [01:52:47] PROBLEM - Packetloss_Average on erbium is CRITICAL: packet_loss_average CRITICAL: 37.5854172414 [01:53:58] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Epic puppet fail [01:55:58] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [01:56:18] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [01:56:47] RECOVERY - Packetloss_Average on erbium is OK: packet_loss_average OKAY: -0.0539559302326 [01:56:58] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [01:56:58] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [01:57:17] PROBLEM - Packetloss_Average on analytics1003 is CRITICAL: packet_loss_average CRITICAL: 14.0737649167 [01:57:46] ok, that's no good, packetloss on analytics is a different DC [01:57:58] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [01:57:58] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:58:27] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:58:58] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [01:59:27] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [01:59:27] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:01:01] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [02:01:01] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:01:17] RECOVERY - Packetloss_Average on analytics1003 is OK: packet_loss_average OKAY: 1.17930608333 [02:01:27] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:01:27] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [02:01:58] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [02:01:58] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [02:01:58] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [02:02:11] neat. [02:02:18] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [02:02:19] seems like a link flap maybe? [02:02:39] the esams case did, analytics1003 is .... strange? [02:02:57] PROBLEM - Packetloss_Average on oxygen is CRITICAL: packet_loss_average CRITICAL: 9.18785825 [02:03:07] ditto that ^ [02:03:27] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [02:03:27] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [02:03:57] from oxygen i can ICMP to something external without loss [02:05:20] yeah I checked lvs boxes too, but analytics100[34] both showing the filter thing [02:05:28] maybe that's normal and we filter those hosts normally, and unrelated [02:06:36] it wasn't on icinga earlier though [02:06:57] RECOVERY - Packetloss_Average on oxygen is OK: packet_loss_average OKAY: 1.15079566667 [02:07:00] I mean maybe packet_loss_average is unrelated to whatever normal circumstances filter pins on analytics [02:07:07] s/pins/pings/ [02:07:17] ah, yea [02:07:46] I'm unclear on what's up we lost the link to esams from eqiad momentarily? [02:07:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [02:09:43] (03PS9) 10BBlack: wikimedia.ee - own zonefile and set external MX [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 (owner: 10Dzahn) [02:10:19] so from analytics1003 you cant tcptraceroute iron, but from iron you can get to ana1003 [02:10:21] chasemp: it looks like we lost some kind of link or routing between esams and eqiad for a few minutes that only affected the private subnet in esams but not the public [02:10:50] possibly-unrelatedly: two hosts in eqiad have temporarily reported packet loss in icinga for a few minutesshortly afterwards [02:11:10] (although perhaps that was picking traffic between those hosts and esams during the esams event?) [02:11:28] hmmm ok thank you, seems weird [02:19:08] (03PS1) 10Aaron Schulz: Removed old jobrunner.ini file [operations/puppet] - 10https://gerrit.wikimedia.org/r/150124 [02:24:12] (03CR) 10BBlack: [C: 032] "I rebased, updating formatting to match wikimedia.com, and validated that the only data diff was the MX part." [operations/dns] - 10https://gerrit.wikimedia.org/r/148762 (owner: 10Dzahn) [02:24:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [02:27:48] (03CR) 10Ori.livneh: [C: 04-1] "Go ahead and just remove the entry for the file rather than ensure => absent it. I'll remove it via Salt from the relevant hosts." [operations/puppet] - 10https://gerrit.wikimedia.org/r/150124 (owner: 10Aaron Schulz) [02:29:27] (03CR) 10BBlack: [C: 032] "Confirmed whitespace-only" [operations/dns] - 10https://gerrit.wikimedia.org/r/150095 (https://bugzilla.wikimedia.org/68769) (owner: 10Dzahn) [02:30:14] (03PS2) 10BBlack: wmftest.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150096 (owner: 10Dzahn) [02:30:41] (03CR) 10BBlack: [C: 032] "Confirmed whitespace-only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150096 (owner: 10Dzahn) [02:31:00] (03PS2) 10BBlack: wikivoyage.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150099 (owner: 10Dzahn) [02:31:35] (03CR) 10BBlack: [C: 032] "Confirmed whitespace only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150099 (owner: 10Dzahn) [02:31:48] (03PS2) 10BBlack: wikiversity.org - retab and align [operations/dns] - 10https://gerrit.wikimedia.org/r/150103 (owner: 10Dzahn) [02:32:25] (03CR) 10BBlack: "Confirmed whitespace only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150103 (owner: 10Dzahn) [02:32:34] (03CR) 10BBlack: [C: 032] "Confirmed whitespace only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150103 (owner: 10Dzahn) [02:32:47] (03PS2) 10BBlack: wikisource.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150105 (owner: 10Dzahn) [02:33:22] (03CR) 10BBlack: [C: 032] "Confirmed whitespace-only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150105 (owner: 10Dzahn) [02:33:31] (03PS2) 10BBlack: wiktionary.org - retab and align [operations/dns] - 10https://gerrit.wikimedia.org/r/150101 (owner: 10Dzahn) [02:34:03] (03CR) 10BBlack: [C: 032] "Confirmed whitespace only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150101 (owner: 10Dzahn) [02:36:21] !log LocalisationUpdate completed (1.24wmf14) at 2014-07-29 02:35:18+00:00 [02:36:28] Logged the message, Master [02:38:20] (03CR) 10BBlack: "Joe is this still a desired change? Is it pending something?" [operations/dns] - 10https://gerrit.wikimedia.org/r/146647 (owner: 10Jkrauska) [02:41:14] <^d> Ugh. https://github.com/elasticsearch/elasticsearch/issues/6168 [02:41:35] (03PS2) 10BBlack: 10.in-addr.arpa - retab only [operations/dns] - 10https://gerrit.wikimedia.org/r/150107 (owner: 10Dzahn) [02:42:08] (03CR) 10BBlack: [C: 032] "Confirmed whitespace-only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150107 (owner: 10Dzahn) [02:43:11] (03PS2) 10BBlack: 174.198.91.in-addr.arpa - retab only [operations/dns] - 10https://gerrit.wikimedia.org/r/150109 (owner: 10Dzahn) [02:43:42] (03CR) 10BBlack: [C: 032] "Confirmed whitespace-only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150109 (owner: 10Dzahn) [02:44:33] (03PS2) 10BBlack: 154.80.208.in-addr.arpa - retab only [operations/dns] - 10https://gerrit.wikimedia.org/r/150108 (owner: 10Dzahn) [02:44:55] (03CR) 10BBlack: [C: 032] "Confirmed whitespace-only." [operations/dns] - 10https://gerrit.wikimedia.org/r/150108 (owner: 10Dzahn) [02:46:09] (03PS2) 10Aaron Schulz: Removed old jobrunner.ini file [operations/puppet] - 10https://gerrit.wikimedia.org/r/150124 [02:48:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [02:59:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [03:10:07] PROBLEM - puppetmaster backend https on strontium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:58] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.026 second response time [03:11:34] !log LocalisationUpdate completed (1.24wmf15) at 2014-07-29 03:10:31+00:00 [03:11:39] Logged the message, Master [03:18:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [03:25:25] (03CR) 10Ori.livneh: [C: 032] "trivial" [operations/puppet] - 10https://gerrit.wikimedia.org/r/150124 (owner: 10Aaron Schulz) [03:48:46] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 29 03:47:39 UTC 2014 (duration 47m 38s) [03:48:51] Logged the message, Master [04:05:28] (03PS1) 10Dzahn: wikibooks.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150127 [04:17:37] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 15 data above and 9 below the confidence bounds [04:17:44] (03PS1) 10Dzahn: wikinews.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150129 [04:19:47] (03PS1) 10Dzahn: wikiquote.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150130 [04:22:09] (03CR) 10Scottlee: [C: 031] wikinews.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150129 (owner: 10Dzahn) [04:22:26] (03CR) 10Scottlee: [C: 031] wikibooks.org - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150127 (owner: 10Dzahn) [04:26:07] (03PS1) 10Dzahn: 1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150132 [04:28:50] (03PS1) 10Dzahn: 26.35.198.in-addr.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150133 [04:33:37] (03PS1) 10Dzahn: 2.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150134 [04:36:29] (03PS1) 10Dzahn: 3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150135 [04:40:10] (03PS1) 10Dzahn: 59.15.185.in-addr.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150136 [04:40:45] (03CR) 10Dzahn: "and that's it. all for Bug 68769" [operations/dns] - 10https://gerrit.wikimedia.org/r/150136 (owner: 10Dzahn) [04:42:07] (03CR) 10Scottlee: [C: 031] 3.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150135 (owner: 10Dzahn) [04:42:36] (03CR) 10Scottlee: [C: 031] 59.15.185.in-addr.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150136 (owner: 10Dzahn) [04:43:50] (03CR) 10Scottlee: [C: 031] 1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150132 (owner: 10Dzahn) [04:44:12] (03CR) 10Scottlee: [C: 031] 26.35.198.in-addr.arpa - retab [operations/dns] - 10https://gerrit.wikimedia.org/r/150133 (owner: 10Dzahn) [05:07:41] apache-config is dead? [05:10:56] long live apache-config [05:13:27] That repo name is probably documented. [05:19:33] (03PS1) 10MZMcBride: Minor tweaks to README_BEFORE_EDITING [operations/apache-config] - 10https://gerrit.wikimedia.org/r/150144 [05:23:27] (03CR) 10Ori.livneh: [C: 032 V: 032] Minor tweaks to README_BEFORE_EDITING [operations/apache-config] - 10https://gerrit.wikimedia.org/r/150144 (owner: 10MZMcBride) [05:23:45] office action. [05:31:20] ori++ [05:36:09] * Carmela tickles ori. [05:39:51] (03PS1) 10Florianschmidtwelzow: WIP: Add Uploadrestriction for Commons in MF [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/150145 (https://bugzilla.wikimedia.org/62598) [05:48:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [05:56:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data above the critical threshold [500.0] [05:57:37] bblack: is that still varnish restarts? [05:57:44] yeah [05:57:59] * ori nods [05:58:41] there's only 100 machines to update manually and watch for random restart failures and then retry repeatedly, one by one with some reasonable delay between them each so the effects don't stack up [05:58:45] it's tons of fun :P [05:59:25] we have 100 varnishes? [05:59:29] yeah [05:59:42] wow, i hadn't realized [06:01:06] i understand a little bit better now why you were motivated to think about varnish orchestration the other week [06:01:45] the main problem with updates and restarts is the random chance of failure due to the stupid persistent mmap implementation [06:01:55] and this update addresses that, so hopefully things will eventually get better [06:02:24] (but after the update + related puppet config change, each and every varnish needs its persistent cache wiped out before the fix takes effect, so that will go out even slower) [06:10:07] PROBLEM - puppetmaster backend https on strontium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:11:07] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 6.258 second response time [06:21:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [06:25:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [06:28:47] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:57] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:57] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:58] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:17] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:17] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:17] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:37] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:37] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:47] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:47] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:17] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:47] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:38:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [06:40:37] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:42:18] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:57] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:45:58] ^ those seem to be a spate of "500 internal server error" on puppetmaster(s) [06:46:17] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:46:17] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:37] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:46:37] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:46:47] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:46:47] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:46:47] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:46:47] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:46:57] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:46:58] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:47:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [06:58:37] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:59:19] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:00:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [07:13:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [07:26:27] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [07:26:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [07:45:26] (03PS1) 10Alexandros Kosiaris: sudo privileges should be in array [operations/puppet] - 10https://gerrit.wikimedia.org/r/150160 [07:51:24] !log uploaded PHP 5.3.10-1ubuntu3.13+wmf1 on apt.wikimedia.org. Puppet will upgrade it across the fleet within 20 mins [07:51:30] Logged the message, Master [07:51:36] let's watch the chaos now :-) [07:53:37] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Tue 29 Jul 2014 05:53:20 UTC [07:57:04] <_joe_> akosiaris: well a minor php update, what could possibl go wrong(TM) [07:59:29] _joe_: exactly :-) [08:20:54] (03PS1) 10Dan-nl: added Universiteits Museum Utrecht to the wgCopyUploadsDomains array [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/150163 [08:26:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [08:33:37] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Tue Jul 29 08:33:28 UTC 2014 [08:37:46] good morning [08:41:44] hey hashar [08:49:04] godog: hi! do you know how serious we are about releasing debian package to the public ? [08:50:13] hashar: you mean a package available from us to be downloaded or uploaded to the official debian archive? [08:52:11] godog: I guess the first step would be to have it offered for download [08:52:21] with the long term goal of having them added to Debian [08:53:11] the reason I am asking is that I really really would like our debian packaging to be as automatic as possible [08:53:29] with a nice workflow and appropriate responsibilities / accesses [08:54:28] hashar_: yep, that's sort of where we are going with the reprepro repo on releases.wm.o I mailed ops about last week [08:54:57] yeah that sounds like a good step forward :D [08:54:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [08:55:14] hashar_: of course the goal as you mentioned would be to have packages uploaded to debian, at least the ones that make sense to have [08:55:29] yup [08:55:40] and on the other side having a Jenkins instance to build the package for us [08:56:04] cause that is totally doable and would let non ops release packages easily :-] [08:58:02] indeed! [09:00:28] !log Zuul bumping Zuul cloner from patchset 21 to patchset 23. Deploying with tag wmf-deploy-2014-07-29-1 [09:00:34] Logged the message, Master [09:02:19] !log restarted zuul-server and zuul-merger on gallium (new version though that is a noop) [09:02:24] Logged the message, Master [09:10:04] <_joe_> !log stopping jobrunner on mw1053, disabling puppet as well - running tests [09:10:09] Logged the message, Master [09:31:57] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [09:37:24] (03CR) 10Giuseppe Lavagetto: [C: 031] HHVM: log warnings and stacktraces [operations/puppet] - 10https://gerrit.wikimedia.org/r/148544 (owner: 10MaxSem) [09:38:40] (03PS5) 10Giuseppe Lavagetto: apache: cherry-pick mods added in Ia46312071 [operations/puppet] - 10https://gerrit.wikimedia.org/r/149066 (owner: 10Ori.livneh) [09:57:57] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [10:29:27] <_joe_> !log temporarily stopping puppet on appservers, releasing a potentially dangerous puppet change [10:29:32] Logged the message, Master [10:29:32] (03CR) 10Giuseppe Lavagetto: [C: 032] apache: cherry-pick mods added in Ia46312071 [operations/puppet] - 10https://gerrit.wikimedia.org/r/149066 (owner: 10Ori.livneh) [10:32:07] (03PS1) 10Matanya: access: Matt Flaschen new key [operations/puppet] - 10https://gerrit.wikimedia.org/r/150175 [10:35:03] <_joe_> !log puppet re-enabled on the appservers [10:35:07]