[00:00:02] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [00:00:02] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:00:02] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [00:00:02] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:00:02] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [00:00:12] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [00:00:12] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [00:00:22] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [00:00:22] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:00:25] Dear Opsen, I need a quick config change in frack. Anyone have rights? [00:00:32] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [00:00:32] RECOVERY - puppet last run on es7 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:00:32] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [00:00:32] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [00:00:32] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:00:33] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [00:00:33] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [00:00:42] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:00:42] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [00:00:52] PROBLEM - Host platinum is DOWN: PING CRITICAL - Packet loss = 100% [00:00:54] oh I hope you all have polarized lenses to filter through the puppetspam [00:01:02] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [00:01:12] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [00:01:12] RECOVERY - Disk space on gallium is OK: DISK OK [00:01:12] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [00:01:12] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:01:12] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [00:01:13] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [00:01:13] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [00:01:14] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [00:01:22] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [00:01:22] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:01:22] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:01:23] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:01:23] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:01:32] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:01:32] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:01:32] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [00:01:32] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [00:01:32] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:01:33] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [00:01:33] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:01:33] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:01:34] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:01:35] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:01:42] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [00:01:42] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:01:42] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [00:01:52] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:01:52] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [00:01:53] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [00:01:53] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [00:02:02] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [00:02:12] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [00:02:12] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:02:12] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:02:12] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [00:02:13] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [00:02:22] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:02:22] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:02:22] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [00:02:23] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:02:32] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [00:02:32] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [00:02:32] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:02:32] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [00:02:32] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:02:33] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:02:33] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [00:02:34] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [00:02:34] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [00:02:35] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [00:02:42] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:02:42] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [00:02:42] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:02:52] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:02:52] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [00:02:52] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [00:08:13] awight: is this fixing an emergency? gerrit ? [00:08:47] mutante: not an emergency. [00:09:00] icinga is back to relatively normal [00:15:23] awight: do you already know if the file is puppetized? [00:22:13] mutante: yes, I just determined that the file is puppetized. Unfortunately, it's in a private repo I don't have access to, so I can't prepare the patchset for u. [00:22:25] * awight makes a plain patch file [00:27:45] (03CR) 10Dzahn: [C: 032] "confirmed key. gpg: Signature made Wed 23 Jul 2014 04:31:32 PM PDT using RSA key ID 9A6B07CC" [operations/puppet] - 10https://gerrit.wikimedia.org/r/144994 (owner: 10JanZerebecki) [00:28:00] (03PS4) 10Dzahn: Give jzerebecki access to analytics data [operations/puppet] - 10https://gerrit.wikimedia.org/r/144994 (owner: 10JanZerebecki) [00:35:45] (03CR) 10Dzahn: [C: 032] Give jzerebecki access to analytics data [operations/puppet] - 10https://gerrit.wikimedia.org/r/144994 (owner: 10JanZerebecki) [00:41:40] spagewmf: you around? [00:41:54] gwicke, was just typinng this... hey, Flow is getting more "503 Service Unavailable" failures contacting Parsoid the last four days. Is parsoid [00:42:06] spagewmf: yes, am also just looking into it [00:42:11] logging its operations omewhere? [00:42:41] the parsoid logs have a lot of api connection failures recently, which might actually be failures to connect to varnish [00:43:15] spagewmf: /var/log/parsoid/parsoid.log on the parsoid boxes [00:43:31] gwicke: thanks! It's not sky-high, but it's affecting browser tests. Do mw.org and test2wiki use different parsoid boxes? [00:43:49] they use the same boxes [00:43:59] and I also see 503s from other wikis in the logs [00:44:25] * spagewmf happy it isn't just Flow :) [00:44:43] !log installing linux-tools on mw1053 to run perf on jobrunner [00:44:44] spagewmf: We have the same problems as well [00:44:49] Logged the message, Master [00:44:51] Flow is only on beta labs, test2wiki, meta-wiki, mw.org, and en-wiki [00:44:56] I'm vaguely inclined to blame HHVM but that's just a gut feeling related to timing [00:45:09] I don't think there's any HHVM in prod yet [00:45:21] I'm leaning towards those Varnishes acting up [00:45:32] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [00:45:41] currently testing that theory by doing requests to them vs. the backend LVS [00:47:44] RoanKattouw: could you check /var/log/varnish on cp1045 & cp1058? [00:49:21] I am occasionally getting 503s when requesting through the varnishes, but so far none when going directly to the Parsoid backend LVS [00:49:35] Looking [00:50:59] RoanKattouw: FYI Flow's "Parsoid 503 Service Unavailable" is bug 68276 [00:51:32] gwicke: /var/log/varnish is an empty directory :S [00:51:45] hm [00:52:03] anything in syslog? [00:53:30] just did another 100 requests each: 2 failures through varnish, none directly through the backend LVS [00:53:42] Nope syslog is clean, just puppet spam in there [00:53:43] also much higher latency through varnish [00:53:47] Maybe there's like a low max connections? [00:54:10] 1024 IIRC [00:54:34] where would varnish log errors when bumping up against that? [00:55:32] No idea [00:55:38] netstat -t -n | wc -l [00:55:40] /var/log/varnish is an empty dir on both boxes [00:55:40] 16369 [00:55:46] on cp1058 [00:55:50] that's suspicious [00:55:51] hah [00:55:55] (03PS1) 10Tim Landscheidt: Tools: Fix and simplify exim redirectors [operations/puppet] - 10https://gerrit.wikimedia.org/r/148917 [00:56:38] I wonder if we changed how long TIME_WAIT lasts on these boxes? [00:57:17] could also be related to the new PHP job runner [00:57:31] I see !15k open connections on cp1045, of which ~10k are TIME_WAIT [00:57:41] Ooh right [00:57:54] maybe that closes connections less quickly [01:01:09] (03PS1) 10Spage: Remove duplicate $wgFlowParsoidURL setting. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148918 [01:02:34] (03CR) 10Tim Landscheidt: "Tested hot on tools-mail with "ln -f localuser.scfc localuser; exim -bt $ADDRESS; ln -f localuser.bak localuser" & dito for maintainers." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148917 (owner: 10Tim Landscheidt) [01:07:20] RoanKattouw: I also see exactly 1000 connections with 10.2.2.28, the LVS backend [01:07:53] just saw 1001 [01:08:07] might be limited to 1024 [01:08:11] Hmm right [01:08:24] But at the same time, we have like 30k open conns to the Varnish boxes [01:08:33] So it sounds like there's a bottleneck effect going on there [01:08:44] (of which about 20k are TIME_WAIT, but still, that leaves 10k) [01:09:17] a lot of those connections are internal, between front & back-end [01:09:43] just saw up to 5k each from the respective front-ends on cp1058 [01:09:55] netstat -t -n | awk '{print $5;}' | sed s/:.*$// | sort | uniq -c | sort -n [01:11:28] spikes up to 6k each [01:14:01] RoanKattouw: https://gist.github.com/gwicke/0ad7b319d08af294ad68 [01:15:23] Oh, right [01:15:26] That makes sense [01:15:57] And exactly 1000 to LVS [01:16:02] gwicke, what is that an output of? # of connections per ip? [01:17:18] subbu: yes [01:17:33] on one of the varnish boxes [01:17:47] total number is suspiciously close to 16k [01:18:21] RoanKattouw: could you restart the varnishes? [01:18:32] In a minute [01:18:38] that might at least provide some relief [01:18:48] can then bump the limits in puppet [01:19:46] gwicke, 19489 actually [01:20:12] gwicke: OK I'm gonna restart cp1045, you wanna monitor netstat before and after on that box? [01:20:13] 'max_connections' => 1000, [01:20:23] there we go [01:21:01] RoanKattouw: I think it's pretty likely to be the backend connection limit [01:21:26] Oh well yeah there it is [01:21:33] The new job runners must be more efficient? [01:21:36] Let's bump that number then [01:23:43] RoanKattouw: pushing a change.. [01:25:58] subbu: is there another bug apart from the flow one? [01:26:07] yes, let me find it. [01:26:18] (03PS1) 10GWicke: Bump Parsoid backend connection limit [operations/puppet] - 10https://gerrit.wikimedia.org/r/148924 (https://bugzilla.wikimedia.org/68276) [01:26:26] https://bugzilla.wikimedia.org/show_bug.cgi?id=68405 [01:26:49] (03PS2) 10GWicke: Bump Parsoid backend connection limit [operations/puppet] - 10https://gerrit.wikimedia.org/r/148924 (https://bugzilla.wikimedia.org/68276) [01:26:55] subbu: thx! [01:27:45] RoanKattouw: ^^ [01:28:10] (03PS1) 10Jforrester: Move Mantle dependency to MobileFrontend too, given Ia060b2295 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148925 [01:28:23] (03PS3) 10Catrope: Bump Parsoid backend connection limit [operations/puppet] - 10https://gerrit.wikimedia.org/r/148924 (https://bugzilla.wikimedia.org/68276) (owner: 10GWicke) [01:28:29] (03CR) 10Catrope: [C: 031] Bump Parsoid backend connection limit [operations/puppet] - 10https://gerrit.wikimedia.org/r/148924 (https://bugzilla.wikimedia.org/68276) (owner: 10GWicke) [01:31:55] RoanKattouw, gwicke so why is the connection limit being hit in the last few days? did the request rate to varnish go up or did parsoid get a tad slower for whatever reason? [01:32:10] i guess we don't know the last bit (about parsoid slowing down). [01:32:21] Gabriel said something about a new job runner [01:32:34] so, more requests to varnish [01:32:46] subbu: there's several factors that I could imagine contributing [01:32:53] one is load testing on the pdf renderer [01:33:12] I have not looked into the new job runner yet [01:33:31] if that doesn't do the same forking for the parsoid jobs then connections might be kept open for longer than they used to be [01:35:04] the # of parsoid job runners was also slightly bumped up a few days ago [01:35:26] https://gerrit.wikimedia.org/r/147850 [01:35:38] 18 => 20, rather modest [01:36:39] Oh and apparently we're now running job runners on HHVM [01:36:41] Some of them at lesat [01:36:44] Which would make them faster [01:37:03] maybe, if the other jobs are running faster [01:37:15] for the parsoid jobs it should not make a measurable difference [01:37:27] all they do is a few http requests, which is all C anyway [01:37:45] Right [01:38:22] so, looking at https://gdash.wikimedia.org/dashboards/reqerror/deploys ... the 5xx are all spikey .. but i suppose that is all varnishes, not just parsoid ones [01:38:54] subbu: I think that's only text varnishes [01:38:59] so without parsoid [01:43:25] I guess all roots are out right now [01:44:47] ocg render requests: https://graphite.wikimedia.org/render/?width=1613&height=750&_salt=1406166262.046&target=ocg.pdf.frontend.requests.render.count&target=ocg.pdf.frontend.requests.render.rate&from=-1weeks [01:45:13] (03CR) 10Spage: [C: 032] "Makes sense, and since https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/label=deployment-bastion-eqiad,wikidb=enwikine" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148925 (owner: 10Jforrester) [01:45:22] (03Merged) 10jenkins-bot: Move Mantle dependency to MobileFrontend too, given Ia060b2295 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148925 (owner: 10Jforrester) [01:45:43] spagewmf: Argh, did you mean to do that? [01:46:21] spagewmf: You're not supposed to +2 things in mediawiki-config unless you're about to deploy them to production immediately [01:46:24] spagewmf: I was going to schedule it for tomorrow afternoon. Unannounced out of process config changes considered harmful. [01:46:40] Now the repo and the deployment are out of sync [01:46:42] James_F: you're right, shoulda +1d [01:46:50] sorry [01:47:44] subbu: definitely looks like a likely contributor [01:47:46] you need something for beta labs though? Could put it in CommonSettings-labs.php. [01:48:17] (03PS1) 10Yurik: LABS: Better font for zero gif banners [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148928 [01:48:30] gwicke, yes, the 500 spike since y'day corresponds to that. [01:48:58] (03CR) 10Yurik: [C: 032] LABS: Better font for zero gif banners [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148928 (owner: 10Yurik) [01:49:05] (03Merged) 10jenkins-bot: LABS: Better font for zero gif banners [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148928 (owner: 10Yurik) [01:54:49] (03PS1) 10Spage: Revert "Move Mantle dependency to MobileFrontend too, given Ia060b2295" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148931 [01:55:22] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 3 unmerged changes in mediawiki_config (dir /a/common/). [01:55:57] Hm.. I got a couple 503's just now from Parsoid when opening VE on mediawiki.org [01:56:00] took a few retries before it worked [01:56:02] subbu: [01:56:18] Krinkle: We've already figured out what's wrong [01:56:21] k [01:56:25] We need a puppet change to get merged though [01:56:30] It wasn't plugged in? [01:56:32] haha [01:56:38] No, max_connections was set to 100 [01:56:40] *1000 [01:56:54] And Gabriel noticed that the number of open connections from Varnish to Parsoid was always exactly 1000 [01:56:58] Ah, one instance getting too much traffic? [01:57:03] As soon as one of those connections ended, a new one took its place [01:57:30] No, just too much traffic in general, being artificially limited to 1000 conns from each Varnish box to the Parsoid cluster [01:57:50] If the number of connections from Varnish to Parsoid is constantly at 1000, that means the limit is too low :) [01:58:04] Right. And one server can handle more than 1000, so we raised the limit? [01:58:13] Or add more servers :P [02:00:04] One server can easily handle more [02:00:13] I don't know why that limit is there, it's probably just the default [02:07:33] (03PS1) 10Yurik: LABS: Playing with zero banner font size [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148935 [02:07:48] (03CR) 10Yurik: [C: 032] LABS: Playing with zero banner font size [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148935 (owner: 10Yurik) [02:09:20] (03Merged) 10jenkins-bot: LABS: Playing with zero banner font size [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148935 (owner: 10Yurik) [02:37:38] !log LocalisationUpdate completed (1.24wmf13) at 2014-07-24 02:36:35+00:00 [02:37:44] Logged the message, Master [02:56:51] subbu: sent a mail to the ops list re merge & deploy [02:56:58] /cc RoanKattouw_away [03:00:04] k, thanks. [03:09:09] !log LocalisationUpdate completed (1.24wmf14) at 2014-07-24 03:08:05+00:00 [03:09:13] Logged the message, Master [03:20:08] (03CR) 10Krinkle: wmflib: add apt_version() (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148512 (owner: 10Ori.livneh) [03:20:12] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 01:19:34 UTC [03:20:43] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Thu Jul 24 03:20:33 UTC 2014 [03:57:39] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 24 03:56:32 UTC 2014 (duration 56m 31s) [03:57:45] Logged the message, Master [04:10:16] (03Abandoned) 10Chad: enwiki gets Cirrus as primary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140755 (owner: 10Chad) [04:10:32] (03Abandoned) 10Chad: Remove "Cirrus as alternative" switches [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140756 (owner: 10Chad) [04:59:09] (03PS1) 10Yurik: Zero banner image font [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148948 [04:59:32] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 1 failures [05:17:32] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:30:42] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:31:32] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.026 second response time [05:44:55] !log labsdb1002 work in progress; it may misbehave. see labs-l for updates [05:45:02] Logged the message, Master [05:56:32] PROBLEM - puppetmaster backend https on strontium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [05:56:32] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 4 failures [05:56:32] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 13 failures [05:56:33] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 19 failures [05:56:33] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 29 failures [05:56:33] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Puppet has 16 failures [05:56:33] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Puppet has 12 failures [05:56:34] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Puppet has 20 failures [05:56:42] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Puppet has 38 failures [05:56:42] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 55 failures [05:56:42] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Puppet has 8 failures [05:56:52] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet has 19 failures [05:56:52] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Puppet has 13 failures [05:56:52] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 40 failures [05:56:52] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: Puppet has 37 failures [05:56:52] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 53 failures [05:56:53] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 12 failures [05:57:02] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 63 failures [05:57:02] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 10 failures [05:57:02] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet has 39 failures [05:57:02] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 47 failures [05:57:12] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 22 failures [05:57:12] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 11 failures [05:57:12] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Puppet has 23 failures [05:57:22] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Puppet has 20 failures [05:57:22] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Puppet has 11 failures [05:57:22] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 52 failures [05:57:22] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 13 failures [05:57:22] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Puppet has 20 failures [05:57:32] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 60 failures [05:57:32] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 16 failures [05:57:32] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet has 19 failures [05:57:32] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 17 failures [05:57:32] PROBLEM - puppet last run on db71 is CRITICAL: CRITICAL: Puppet has 23 failures [05:57:33] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: Puppet has 17 failures [05:57:33] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 23 failures [05:57:34] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Puppet has 61 failures [05:57:34] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 63 failures [05:57:42] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 41 failures [05:57:42] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 50 failures [05:57:42] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 25 failures [05:57:42] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Puppet has 18 failures [05:57:42] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Puppet has 22 failures [05:57:43] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Puppet has 18 failures [05:57:43] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 56 failures [05:57:44] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 63 failures [05:57:44] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 65 failures [05:57:45] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 60 failures [05:57:45] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 58 failures [05:58:38] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Puppet has 25 failures [05:58:38] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 24 failures [05:58:38] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 54 failures [05:58:38] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 15 failures [05:58:38] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Puppet has 18 failures [05:58:38] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 21 failures [05:58:38] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 61 failures [05:58:38] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 66 failures [05:58:38] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 20 failures [05:58:38] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 53 failures [05:58:38] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 52 failures [05:58:38] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 55 failures [05:58:38] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 18 failures [05:58:38] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 19 failures [05:58:38] PROBLEM - puppet last run on db60 is CRITICAL: CRITICAL: Puppet has 23 failures [05:58:38] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 66 failures [05:58:38] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 27 failures [05:58:38] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 13 failures [05:58:38] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 58 failures [05:58:38] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 46 failures [05:58:38] PROBLEM - puppet last run on mexia is CRITICAL: CRITICAL: Puppet has 21 failures [05:58:38] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 24 failures [05:58:38] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 21 failures [05:58:38] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 19 failures [05:58:38] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 23 failures [05:58:38] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 26 failures [05:58:38] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 23 failures [05:58:38] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 53 failures [05:58:38] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 19 failures [05:58:38] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 40 failures [05:58:38] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 19 failures [05:58:38] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 17 failures [05:58:38] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 21 failures [05:58:38] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 21 failures [05:58:38] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 24 failures [05:58:38] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 56 failures [05:58:38] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 20 failures [05:58:38] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 58 failures [05:58:38] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 56 failures [05:58:38] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 69 failures [05:58:38] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 58 failures [05:58:39] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 17 failures [05:58:39] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 23 failures [05:58:42] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 23 failures [05:58:42] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 17 failures [05:58:43] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 59 failures [05:58:43] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 59 failures [05:58:43] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 16 failures [05:58:52] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 59 failures [05:58:52] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 21 failures [05:59:02] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 22 failures [05:59:02] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 22 failures [05:59:12] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 23 failures [05:59:12] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 44 failures [05:59:12] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 53 failures [05:59:22] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 42 failures [05:59:22] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 18 failures [05:59:22] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 53 failures [05:59:22] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 21 failures [05:59:22] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 22 failures [05:59:23] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 23 failures [05:59:23] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 18 failures [05:59:33] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 17 failures [05:59:33] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Puppet has 22 failures [05:59:33] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 17 failures [05:59:33] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 62 failures [05:59:33] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 23 failures [05:59:34] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 19 failures [05:59:34] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 19 failures [05:59:35] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 19 failures [05:59:35] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 19 failures [05:59:36] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 63 failures [05:59:42] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Puppet has 19 failures [05:59:42] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 36 failures [05:59:43] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 22 failures [05:59:43] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 22 failures [05:59:43] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 32 failures [05:59:52] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 53 failures [05:59:52] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 19 failures [05:59:52] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 49 failures [05:59:52] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 49 failures [05:59:52] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 58 failures [05:59:53] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 62 failures [05:59:53] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 23 failures [05:59:54] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 58 failures [05:59:54] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 53 failures [06:00:02] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 53 failures [06:00:02] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 25 failures [06:00:02] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 53 failures [06:00:02] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 65 failures [06:00:02] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 21 failures [06:00:12] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 13 failures [06:00:12] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 59 failures [06:00:13] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 25 failures [06:00:22] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 62 failures [06:00:22] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 54 failures [06:00:22] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Puppet has 21 failures [06:00:22] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 25 failures [06:00:22] PROBLEM - puppet last run on tantalum is CRITICAL: CRITICAL: Puppet has 19 failures [06:00:32] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Puppet has 54 failures [06:00:32] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 16 failures [06:00:32] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 58 failures [06:00:33] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 14 failures [06:00:33] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 46 failures [06:00:33] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 26 failures [06:00:33] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 18 failures [06:00:34] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 23 failures [06:00:34] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 63 failures [06:00:35] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 23 failures [06:00:35] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 27 failures [06:00:36] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 59 failures [06:00:36] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 19 failures [06:00:42] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 52 failures [06:00:42] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 19 failures [06:00:42] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 53 failures [06:00:52] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 62 failures [06:00:52] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 45 failures [06:00:53] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Puppet has 43 failures [06:00:53] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 18 failures [06:00:53] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 60 failures [06:00:53] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 19 failures [06:01:12] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 23 failures [06:01:12] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 17 failures [06:01:13] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 23 failures [06:01:13] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: Puppet has 22 failures [06:01:22] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Puppet has 40 failures [06:01:22] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 56 failures [06:01:22] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 19 failures [06:01:32] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 58 failures [06:01:32] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 19 failures [06:01:32] PROBLEM - puppet last run on es7 is CRITICAL: CRITICAL: Puppet has 23 failures [06:01:33] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 43 failures [06:01:33] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 55 failures [06:01:33] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 65 failures [06:01:33] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 23 failures [06:01:42] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 25 failures [06:01:42] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 40 failures [06:01:42] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 40 failures [06:01:42] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 23 failures [06:01:52] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 61 failures [06:01:52] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 20 failures [06:02:02] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 16 failures [06:02:02] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Puppet has 16 failures [06:02:02] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 58 failures [06:02:02] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 16 failures [06:02:02] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 24 failures [06:02:12] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 20 failures [06:02:12] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 23 failures [06:02:22] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 58 failures [06:02:22] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 26 failures [06:02:22] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 26 failures [06:02:32] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 58 failures [06:02:32] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 23 failures [06:02:32] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Puppet has 18 failures [06:02:32] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 20 failures [06:02:33] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 18 failures [06:02:33] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 22 failures [06:02:33] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: Puppet has 42 failures [06:02:34] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 57 failures [06:02:34] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 18 failures [06:02:35] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 17 failures [06:02:35] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet has 53 failures [06:02:42] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 14 failures [06:02:42] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Puppet has 17 failures [06:02:52] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 58 failures [06:02:52] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 20 failures [06:03:02] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Puppet has 54 failures [06:03:12] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 24 failures [06:03:12] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 19 failures [06:03:12] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Puppet has 61 failures [06:03:13] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Puppet has 31 failures [06:03:13] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 17 failures [06:03:13] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Puppet has 60 failures [06:03:13] PROBLEM - puppet last run on tarin is CRITICAL: CRITICAL: Puppet has 16 failures [06:03:14] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 54 failures [06:03:14] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 24 failures [06:03:15] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 19 failures [06:03:23] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Puppet has 17 failures [06:03:24] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Puppet has 62 failures [06:03:24] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 23 failures [06:03:24] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Puppet has 25 failures [06:03:24] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 27 failures [06:03:32] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Puppet has 17 failures [06:03:33] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 18 failures [06:03:33] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 20 failures [06:03:33] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 21 failures [06:03:33] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 23 failures [06:03:33] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 61 failures [06:03:33] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Puppet has 17 failures [06:03:34] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet has 57 failures [06:03:34] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Puppet has 50 failures [06:03:35] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Puppet has 26 failures [06:03:35] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Puppet has 21 failures [06:03:36] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 58 failures [06:03:36] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 28 failures [06:03:42] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 22 failures [06:03:42] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Puppet has 47 failures [06:03:52] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 20 failures [06:03:52] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet has 28 failures [06:04:02] PROBLEM - puppet last run on db1017 is CRITICAL: CRITICAL: Puppet has 19 failures [06:04:02] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet has 23 failures [06:04:12] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 49 failures [06:04:12] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 46 failures [06:04:12] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Puppet has 49 failures [06:04:12] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 55 failures [06:04:12] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 28 failures [06:04:22] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 22 failures [06:04:22] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 14 failures [06:04:22] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 19 failures [06:04:22] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 19 failures [06:04:23] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Puppet has 20 failures [06:04:32] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet has 23 failures [06:04:32] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 60 failures [06:04:32] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 20 failures [06:04:32] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 21 failures [06:04:32] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 21 failures [06:04:33] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 15 failures [06:04:33] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 21 failures [06:04:34] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 57 failures [06:04:34] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 54 failures [06:04:35] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 21 failures [06:04:35] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 33 failures [06:04:36] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 52 failures [06:04:36] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 18 failures [06:04:37] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet has 20 failures [06:04:37] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 60 failures [06:04:38] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 22 failures [06:04:42] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Puppet has 62 failures [06:04:42] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: Puppet has 39 failures [06:04:42] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 31 failures [06:04:43] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 67 failures [06:04:43] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 13 failures [06:04:43] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 54 failures [06:04:43] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 55 failures [06:04:52] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 16 failures [06:04:52] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet has 56 failures [06:04:52] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 52 failures [06:04:52] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 18 failures [06:04:52] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 51 failures [06:05:02] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 53 failures [06:05:12] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet has 23 failures [06:05:12] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 31 failures [06:05:12] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 20 failures [06:05:12] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: Puppet has 17 failures [06:05:12] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 59 failures [06:05:13] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 67 failures [06:05:13] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet has 70 failures [06:05:14] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 52 failures [06:05:14] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 23 failures [06:05:22] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 21 failures [06:05:22] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet has 21 failures [06:05:22] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 25 failures [06:05:22] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 65 failures [06:05:22] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Puppet has 21 failures [06:05:32] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: Puppet has 18 failures [06:05:32] PROBLEM - puppet last run on db73 is CRITICAL: CRITICAL: Puppet has 20 failures [06:05:32] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 21 failures [06:05:33] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Puppet has 22 failures [06:05:33] PROBLEM - puppet last run on db74 is CRITICAL: CRITICAL: Puppet has 26 failures [06:05:42] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 17 failures [06:05:42] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Puppet has 17 failures [06:05:42] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Puppet has 20 failures [06:05:42] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet has 59 failures [06:05:42] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 41 failures [06:05:43] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Puppet has 17 failures [06:05:52] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 13 failures [06:05:52] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: Puppet has 40 failures [06:05:52] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 69 failures [06:05:52] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 24 failures [06:05:52] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 19 failures [06:05:53] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 22 failures [06:05:53] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 50 failures [06:05:54] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 17 failures [06:06:02] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 18 failures [06:06:02] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 19 failures [06:06:12] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 59 failures [06:06:12] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Puppet has 32 failures [06:06:13] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 19 failures [06:06:13] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 22 failures [06:06:13] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet has 68 failures [06:06:22] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 51 failures [06:06:22] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 16 failures [06:06:22] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 12 failures [06:06:22] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Puppet has 16 failures [06:06:22] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 15 failures [06:06:23] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 62 failures [06:06:23] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Puppet has 18 failures [06:06:24] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Puppet has 22 failures [06:06:32] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 58 failures [06:06:32] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 20 failures [06:06:32] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 20 failures [06:06:32] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 22 failures [06:06:32] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 24 failures [06:06:33] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Puppet has 20 failures [06:06:33] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 66 failures [06:06:34] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 56 failures [06:06:34] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 18 failures [06:06:35] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 21 failures [06:06:42] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 49 failures [06:06:42] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Puppet has 17 failures [06:06:42] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 28 failures [06:06:42] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 20 failures [06:06:42] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 50 failures [06:06:52] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 15 failures [06:06:52] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 18 failures [06:07:02] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 55 failures [06:07:12] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 18 failures [06:07:13] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 14 failures [06:07:13] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Puppet has 20 failures [06:07:13] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 18 failures [06:07:13] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 59 failures [06:07:13] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 64 failures [06:07:13] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 57 failures [06:07:14] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 47 failures [06:07:14] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Puppet has 20 failures [06:07:15] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 42 failures [06:07:15] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 58 failures [06:07:22] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Puppet has 23 failures [06:07:22] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 23 failures [06:07:22] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 58 failures [06:07:32] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 47 failures [06:07:33] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 43 failures [06:07:33] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 57 failures [06:07:33] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 64 failures [06:07:33] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 22 failures [06:07:33] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 16 failures [06:07:33] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 21 failures [06:07:34] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Puppet has 20 failures [06:07:34] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 21 failures [06:07:35] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 27 failures [06:07:35] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Puppet has 48 failures [06:07:36] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 16 failures [06:07:36] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 22 failures [06:07:42] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 52 failures [06:07:42] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 56 failures [06:07:42] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 50 failures [06:07:42] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 61 failures [06:07:42] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 63 failures [06:07:43] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 25 failures [06:07:43] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 55 failures [06:07:52] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 50 failures [06:07:52] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 21 failures [06:07:52] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 57 failures [06:08:02] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 61 failures [06:08:13] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 60 failures [06:08:13] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 57 failures [06:08:13] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 19 failures [06:08:13] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 50 failures [06:08:13] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 59 failures [06:08:22] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 19 failures [06:08:22] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 47 failures [06:08:22] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 52 failures [06:08:22] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 58 failures [06:08:22] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 17 failures [06:08:23] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 47 failures [06:08:23] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 65 failures [06:08:32] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 22 failures [06:08:33] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 19 failures [06:08:33] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 26 failures [06:08:33] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 21 failures [06:08:33] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 21 failures [06:08:33] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 16 failures [06:08:42] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet has 62 failures [06:08:42] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 31 failures [06:08:42] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 24 failures [06:08:42] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 19 failures [06:08:43] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 34 failures [06:08:43] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 43 failures [06:08:43] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 62 failures [06:08:44] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 57 failures [06:08:52] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 22 failures [06:08:52] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 19 failures [06:08:52] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 56 failures [06:09:02] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 21 failures [06:09:12] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 62 failures [06:09:12] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 49 failures [06:09:12] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet has 46 failures [06:09:13] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet has 20 failures [06:09:13] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: Puppet has 30 failures [06:09:13] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 44 failures [06:09:13] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 21 failures [06:09:14] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 55 failures [06:09:22] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 22 failures [06:09:22] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 18 failures [06:09:22] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 26 failures [06:09:22] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 55 failures [06:09:22] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 18 failures [06:09:32] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 21 failures [06:09:33] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 19 failures [06:09:33] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: Puppet has 22 failures [06:09:33] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 20 failures [06:09:33] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 17 failures [06:09:33] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 14 failures [06:09:33] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 20 failures [06:09:34] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 21 failures [06:09:34] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 19 failures [06:09:35] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 18 failures [06:09:35] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Puppet has 57 failures [06:09:36] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 66 failures [06:09:36] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 44 failures [06:09:37] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 15 failures [06:09:42] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 19 failures [06:09:42] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 25 failures [06:09:42] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Puppet has 18 failures [06:09:43] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 24 failures [06:09:43] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 62 failures [06:09:52] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 22 failures [06:09:52] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 22 failures [06:09:52] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Puppet has 20 failures [06:09:52] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 22 failures [06:09:52] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Puppet has 17 failures [06:10:02] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet has 27 failures [06:10:02] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 61 failures [06:10:12] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 57 failures [06:10:12] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 30 failures [06:10:12] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Puppet has 45 failures [06:10:12] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Puppet has 21 failures [06:10:22] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet has 17 failures [06:10:22] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Puppet has 22 failures [06:10:22] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Puppet has 61 failures [06:10:22] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 59 failures [06:10:22] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Puppet has 23 failures [06:10:23] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Puppet has 19 failures [06:10:32] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Puppet has 21 failures [06:10:32] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Puppet has 14 failures [06:10:32] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 18 failures [06:10:33] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Puppet has 52 failures [06:10:33] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 17 failures [06:10:33] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Puppet has 56 failures [06:10:33] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Puppet has 17 failures [06:10:34] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Puppet has 26 failures [06:10:42] PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Puppet has 30 failures [06:10:42] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 65 failures [06:10:42] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Puppet has 50 failures [06:10:42] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Puppet has 27 failures [06:10:52] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 52 failures [06:10:52] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 56 failures [06:10:52] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Puppet has 25 failures [06:11:02] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Puppet has 53 failures [06:11:12] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Puppet has 24 failures [06:11:12] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Puppet has 60 failures [06:11:12] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Puppet has 16 failures [06:11:12] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 28 failures [06:11:13] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: Puppet has 37 failures [06:11:22] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Puppet has 24 failures [06:11:22] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 23 failures [06:11:22] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Puppet has 23 failures [06:11:22] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 48 failures [06:11:22] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 24 failures [06:11:23] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Puppet has 23 failures [06:11:23] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Puppet has 59 failures [06:11:24] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: Puppet has 16 failures [06:11:33] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 61 failures [06:11:33] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 12 failures [06:11:33] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: Puppet has 16 failures [06:11:33] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet has 56 failures [06:11:33] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 17 failures [06:11:33] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: Puppet has 22 failures [06:11:33] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 21 failures [06:11:34] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 22 failures [06:11:34] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Puppet has 22 failures [06:11:35] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 20 failures [06:11:35] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Puppet has 20 failures [06:11:36] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Puppet has 20 failures [06:11:36] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 56 failures [06:11:37] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 57 failures [06:11:37] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 32 failures [06:11:38] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: Puppet has 20 failures [06:11:38] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 23 failures [06:11:39] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet has 59 failures [06:11:39] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Puppet has 50 failures [06:11:40] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 59 failures [06:11:40] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 52 failures [06:11:41] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Puppet has 24 failures [06:11:41] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: Puppet has 20 failures [06:11:42] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Puppet has 21 failures [06:11:42] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: Puppet has 20 failures [06:11:43] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: Puppet has 47 failures [06:11:43] PROBLEM - puppet last run on db69 is CRITICAL: CRITICAL: Puppet has 20 failures [06:11:44] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 24 failures [06:11:44] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: Puppet has 18 failures [06:11:45] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 13 failures [06:11:52] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Puppet has 19 failures [06:11:52] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 20 failures [06:11:52] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: Puppet has 23 failures [06:11:52] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet has 58 failures [06:12:02] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet has 22 failures [06:12:02] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Puppet has 16 failures [06:12:02] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Puppet has 21 failures [06:12:12] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Puppet has 17 failures [06:12:13] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: Puppet has 30 failures [06:12:13] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 51 failures [06:12:22] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 21 failures [06:12:22] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet has 16 failures [06:12:22] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 52 failures [06:12:22] PROBLEM - puppet last run on db72 is CRITICAL: CRITICAL: Puppet has 21 failures [06:12:23] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Puppet has 27 failures [06:12:23] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 20 failures [06:12:23] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Puppet has 63 failures [06:12:24] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 62 failures [06:12:24] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: Puppet has 49 failures [06:12:25] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 21 failures [06:12:33] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 22 failures [06:12:33] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 18 failures [06:12:33] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 22 failures [06:12:33] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 22 failures [06:12:33] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 23 failures [06:12:33] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Puppet has 17 failures [06:12:33] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet has 63 failures [06:12:34] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Puppet has 18 failures [06:12:34] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Puppet has 58 failures [06:12:35] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Puppet has 26 failures [06:12:35] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 18 failures [06:12:36] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Puppet has 57 failures [06:12:36] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Puppet has 21 failures [06:12:37] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Puppet has 18 failures [06:12:42] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Puppet has 21 failures [06:12:42] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Puppet has 23 failures [06:12:52] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Puppet has 62 failures [06:12:52] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Puppet has 51 failures [06:12:52] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: Puppet has 20 failures [06:12:53] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Puppet has 24 failures [06:13:02] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Puppet has 16 failures [06:13:02] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Puppet has 60 failures [06:13:02] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 62 failures [06:13:12] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Puppet has 23 failures [06:13:12] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Puppet has 23 failures [06:13:12] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 53 failures [06:13:12] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet has 59 failures [06:13:12] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet has 49 failures [06:13:13] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Puppet has 36 failures [06:13:13] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 19 failures [06:13:14] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 49 failures [06:13:14] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Puppet has 54 failures [06:13:15] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Puppet has 56 failures [06:13:15] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Puppet has 16 failures [06:13:16] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: Puppet has 21 failures [06:13:16] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 60 failures [06:13:17] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: Puppet has 41 failures [06:13:22] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Puppet has 18 failures [06:13:22] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 55 failures [06:13:22] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Puppet has 60 failures [06:13:22] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet has 61 failures [06:13:32] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Puppet has 56 failures [06:13:32] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Puppet has 21 failures [06:13:32] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Puppet has 47 failures [06:13:32] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 22 failures [06:13:33] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: Puppet has 44 failures [06:13:33] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: Puppet has 25 failures [06:13:42] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Puppet has 18 failures [06:13:42] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Puppet has 55 failures [06:13:42] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 20 failures [06:13:42] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: Puppet has 48 failures [06:13:42] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet has 20 failures [06:13:43] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: Puppet has 63 failures [06:13:43] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Puppet has 19 failures [06:13:44] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Puppet has 24 failures [06:13:44] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Puppet has 21 failures [06:13:45] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 21 failures [06:13:45] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Puppet has 16 failures [06:13:46] PROBLEM - puppet last run on ssl1008 is CRITICAL: CRITICAL: Puppet has 29 failures [06:13:52] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Puppet has 19 failures [06:13:52] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 56 failures [06:13:53] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Puppet has 23 failures [06:14:02] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 60 failures [06:14:02] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 22 failures [06:14:02] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 25 failures [06:14:12] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: Puppet has 14 failures [06:14:13] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet has 59 failures [06:14:13] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Puppet has 22 failures [06:14:13] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Puppet has 24 failures [06:14:22] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 19 failures [06:14:32] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet has 25 failures [06:14:33] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 56 failures [06:14:33] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 27 failures [06:14:33] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Puppet has 61 failures [06:14:33] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: Puppet has 18 failures [06:14:33] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 21 failures [06:14:33] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: Puppet has 18 failures [06:14:34] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Puppet has 55 failures [06:14:34] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: Puppet has 17 failures [06:14:35] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Puppet has 20 failures [06:14:35] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: Puppet has 24 failures [06:14:36] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Puppet has 24 failures [06:14:42] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 18 failures [06:14:42] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Puppet has 23 failures [06:14:42] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: Puppet has 19 failures [06:14:42] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Puppet has 21 failures [06:14:52] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 18 failures [06:14:52] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Puppet has 21 failures [06:14:52] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 52 failures [06:15:02] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 55 failures [06:15:02] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: Puppet has 12 failures [06:15:02] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 24 failures [06:15:02] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Puppet has 61 failures [06:15:02] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet has 60 failures [06:15:03] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 47 failures [06:15:12] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 50 failures [06:15:12] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Puppet has 56 failures [06:15:13] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 20 failures [06:15:13] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Puppet has 22 failures [06:15:13] PROBLEM - puppet last run on search1013 is CRITICAL: CRITICAL: Puppet has 42 failures [06:15:13] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 20 failures [06:15:13] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Puppet has 16 failures [06:15:22] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet has 18 failures [06:15:22] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Puppet has 56 failures [06:15:22] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Puppet has 23 failures [06:15:32] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Puppet has 19 failures [06:15:32] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 18 failures [06:15:32] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet has 52 failures [06:15:33] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Puppet has 23 failures [06:15:42] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 61 failures [06:15:42] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet has 61 failures [06:15:42] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet has 66 failures [06:15:42] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 20 failures [06:15:52] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 57 failures [06:15:52] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 67 failures [06:15:52] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 25 failures [06:16:02] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 23 failures [06:16:02] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 49 failures [06:16:13] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 27 failures [06:16:13] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Puppet has 21 failures [06:16:13] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 17 failures [06:16:22] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: Puppet has 28 failures [06:16:22] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 62 failures [06:16:23] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Puppet has 17 failures [06:16:23] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: Puppet has 43 failures [06:16:42] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 53 failures [06:17:02] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Epic puppet fail [06:20:52] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Epic puppet fail [06:25:33] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 5.363 second response time [06:26:33] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:26:42] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:26:42] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:26:42] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:26:52] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:26:53] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:27:02] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:27:02] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:27:12] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:27:12] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:27:12] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:27:12] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:27:13] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:27:13] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:27:13] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:27:14] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:27:22] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:27:22] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:27:22] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:27:22] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:27:22] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:27:23] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:27:23] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:27:33] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:27:33] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:27:33] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:27:33] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:27:33] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:27:33] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:27:33] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:27:34] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:27:34] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:27:42] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:27:42] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:27:42] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:27:42] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:27:42] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:27:43] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:27:43] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:27:44] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:27:52] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:27:52] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:27:52] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:27:52] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:27:52] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:28:02] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:28:02] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:28:12] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:28:13] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:28:13] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:28:13] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:28:13] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:28:22] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:28:22] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:28:22] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:28:22] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:28:22] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:28:23] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:28:23] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:28:32] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:28:32] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:28:32] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:28:32] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:28:32] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:28:33] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:28:33] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:28:34] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:28:34] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:28:35] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:28:35] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:28:36] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:28:36] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:28:37] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:28:37] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:28:38] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:28:38] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:28:39] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:28:39] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:28:40] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:28:42] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:28:42] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:28:42] RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:28:42] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:28:42] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:28:43] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:28:43] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:28:44] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:28:52] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:28:52] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:28:52] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:28:52] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:29:12] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:29:12] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:29:12] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:29:12] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:29:13] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:29:22] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:29:22] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:29:22] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:29:22] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:29:22] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:29:23] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:29:23] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:29:24] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:29:32] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:29:32] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:29:32] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:29:32] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:29:33] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:29:33] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:29:33] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:29:34] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:29:34] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:29:35] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:29:35] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:29:36] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:29:36] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:29:37] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:29:37] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:29:38] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:29:42] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:29:42] RECOVERY - puppet last run on db69 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:29:42] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:29:42] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:29:52] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:29:52] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:29:52] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:29:52] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:29:52] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:29:53] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:29:53] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:30:02] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:30:02] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:30:02] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:30:02] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:30:12] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:30:12] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:30:12] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:30:12] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:30:12] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:30:13] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:30:13] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:30:14] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:30:22] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:30:22] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:30:22] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:30:22] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:30:22] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:30:23] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:30:32] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:30:32] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:30:32] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:30:33] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:30:33] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:30:33] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:30:33] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:30:34] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:30:34] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:30:35] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:30:35] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:30:36] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:30:36] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:30:37] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:30:37] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:30:38] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:30:42] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:30:42] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:31:02] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:31:02] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:31:13] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:31:13] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:31:13] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:31:13] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:31:13] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:31:14] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:31:14] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:31:15] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:31:15] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:31:16] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:31:22] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:31:22] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:31:22] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:31:22] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:31:22] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:31:23] RECOVERY - puppet last run on db72 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:31:23] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:31:24] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:31:24] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:31:25] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:31:32] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:31:33] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:31:33] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:31:33] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:31:33] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:31:33] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:31:33] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:31:34] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:31:34] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:31:42] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:31:42] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:31:42] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:31:42] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:31:42] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:31:43] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:31:43] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:31:44] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:31:44] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:31:45] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:31:45] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:31:46] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:31:52] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:31:52] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:31:52] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:31:53] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:31:53] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:32:02] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:32:02] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:32:02] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:32:12] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:32:12] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:32:12] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:32:12] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:32:12] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:32:13] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:32:22] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:32:32] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:32:32] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:32:32] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:32:32] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:32:32] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:32:33] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:32:33] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:32:34] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:32:34] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:32:35] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:32:35] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:32:42] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:32:42] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:32:42] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:32:42] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:32:42] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:32:52] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:32:52] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:32:53] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:33:07] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:33:08] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:33:08] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:33:24] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:33:24] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:33:32] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:33:33] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:33:33] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:33:33] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [06:33:33] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:33:33] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:33:33] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:33:34] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:33:42] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:33:42] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:33:42] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:33:42] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:33:42] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:33:43] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:33:43] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:33:44] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:33:44] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:33:52] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:33:53] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:33:53] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:34:02] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:34:03] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:34:03] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:34:22] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:34:22] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:34:22] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:34:32] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:34:32] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:34:32] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:34:33] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:34:33] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:34:42] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:34:42] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:34:42] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:34:42] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:34:42] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:34:43] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:34:43] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:34:52] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:34:52] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:34:52] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:34:53] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:34:53] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:34:53] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:34:53] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:35:02] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:35:02] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:35:02] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:35:02] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:35:02] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:35:03] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:35:03] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:35:12] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:35:12] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:35:12] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:35:22] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:35:22] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:35:22] RECOVERY - puppet last run on db60 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:35:32] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:35:32] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:35:32] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:35:32] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:35:32] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:35:33] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:35:33] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:35:34] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:35:34] RECOVERY - puppet last run on db71 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:35:35] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:35:42] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:35:42] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:35:42] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:35:42] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:35:42] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:35:43] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:35:43] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:35:44] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:35:44] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:35:45] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:35:45] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:35:46] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:35:46] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:35:47] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:35:47] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:35:52] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:35:53] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:35:53] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:36:02] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:36:02] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:36:02] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:36:02] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:36:12] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:36:12] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:36:12] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:36:13] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:36:22] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:36:22] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:36:22] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:36:22] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:36:22] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:36:23] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:36:32] RECOVERY - puppet last run on mexia is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:36:32] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:36:32] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:36:32] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:36:32] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:36:33] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:36:33] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:36:34] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:36:34] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:36:35] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:36:35] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:36:36] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:36:36] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:36:37] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:36:37] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:36:38] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:36:42] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:36:42] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:36:42] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:36:43] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:36:43] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:36:43] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:36:52] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:36:52] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:36:52] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:36:52] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:36:53] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:36:53] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:36:53] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:37:02] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:37:12] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:37:12] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:37:12] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:37:22] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:37:22] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:37:32] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:37:32] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:37:32] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:37:32] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:37:32] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:37:33] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:37:33] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:37:34] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:37:34] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:37:35] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:37:35] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:37:42] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:37:42] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:37:42] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:37:42] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:37:42] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:37:43] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:37:43] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:37:44] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:37:52] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:37:52] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:37:52] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:37:52] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:37:52] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:37:53] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:37:53] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:37:54] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:38:02] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:38:02] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:38:02] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:38:02] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:38:02] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:38:12] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:38:12] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:38:12] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:38:22] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:38:23] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:38:23] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:38:23] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:38:32] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:38:33] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:38:33] RECOVERY - puppet last run on tantalum is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:38:33] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:38:33] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:38:33] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:38:33] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:38:34] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:38:34] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:38:35] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:38:42] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:38:42] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:38:52] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:38:52] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:38:53] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:38:53] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:39:02] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:39:12] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:39:22] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:39:22] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:39:32] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:39:32] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:39:32] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:39:32] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:39:32] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:39:33] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:39:33] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:39:34] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:39:34] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:39:35] RECOVERY - puppet last run on es7 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:39:35] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:39:36] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:39:42] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:39:42] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:39:43] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:39:43] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:39:43] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:39:43] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:39:43] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:39:52] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:39:52] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:39:52] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:39:52] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:39:52] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:39:53] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:39:53] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:39:54] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:39:54] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:39:55] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:40:02] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:40:02] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:40:02] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:40:02] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:40:02] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:40:12] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:40:22] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:40:22] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:40:22] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:40:32] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:40:32] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:40:32] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:40:33] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:40:33] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:40:33] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:40:33] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:40:34] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:40:34] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:40:35] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:40:42] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:40:42] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:40:42] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:40:42] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:40:43] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:40:43] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:40:52] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:40:52] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:40:52] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:40:52] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:41:02] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:41:12] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:41:12] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:41:13] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:41:22] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:41:22] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:41:22] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:41:22] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:41:32] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:41:32] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:41:32] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:41:33] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:41:33] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:41:33] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:41:33] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:41:34] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:41:42] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:41:42] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:41:42] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:41:42] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:41:42] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:41:52] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:41:52] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:41:52] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:42:03] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:42:12] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:42:13] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:42:22] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:42:32] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:42:32] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:42:32] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:42:32] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:42:33] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:42:33] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:42:33] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:42:34] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:42:34] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:42:35] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:42:43] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:42:43] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:42:43] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:42:43] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:42:43] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:42:43] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:42:43] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:42:44] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:42:44] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:42:52] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:42:52] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:42:52] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:42:52] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:42:52] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:42:53] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:42:53] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:42:54] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:43:12] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:43:12] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:43:12] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:43:12] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:43:12] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:43:13] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:43:13] RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:43:22] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:43:22] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:43:22] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:43:32] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:43:32] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:43:32] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:43:32] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:43:32] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:43:33] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:43:33] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:43:34] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:43:42] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:43:42] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:43:42] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:43:42] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:43:42] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:43:43] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:43:43] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:43:44] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:43:44] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:43:52] RECOVERY - puppet last run on db74 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:43:52] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:43:52] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:43:52] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:43:52] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:43:53] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:43:53] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:43:54] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:43:54] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:43:55] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:43:55] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:43:56] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:44:02] RECOVERY - puppet last run on db1017 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:44:02] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:44:02] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:44:02] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:44:12] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:44:12] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:44:12] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:44:12] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:44:12] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:44:13] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:44:13] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:44:22] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:44:22] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:44:22] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:44:22] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:44:22] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:44:23] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:44:32] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:44:32] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:44:32] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:44:32] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:44:32] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:44:33] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:44:33] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:44:34] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:44:34] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:44:35] RECOVERY - puppet last run on db73 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:44:35] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:44:36] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:44:36] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:44:37] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:44:37] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:44:38] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:44:42] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:44:42] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:44:42] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:44:43] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:44:43] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:44:43] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:44:43] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:44:44] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:44:44] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:44:52] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:44:52] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:45:12] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:45:22] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:45:22] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:45:22] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:45:22] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:45:22] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:45:32] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:45:32] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:45:33] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:45:33] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:45:42] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:45:42] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:45:42] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:45:43] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:45:43] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:46:02] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:46:22] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:46:22] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:46:32] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:46:32] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:46:32] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:32] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:34] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:46:42] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:46:42] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:52] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:47:22] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:47:52] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:02] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:56:22] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 1 failures [07:05:52] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:14:22] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:51:10] akosiaris: hi! yesterday I told you about some ferm rules needed for Zuul merger process on different boxes. I wanted to use puppet collection but you recommended using network.pp . I believe I have a working patch for it https://gerrit.wikimedia.org/r/#/c/148640/ :-D [08:51:29] no urgency though . Labs instance can't access lanthanum anyway (production box with private IP) [08:56:29] (03CR) 10Hashar: "Ah puppet. The puppet catalog compiler complains ( http://puppet-compiler.wmflabs.org/179/change/14 )" [operations/puppet] - 10https://gerrit.wikimedia.org/r/141758 (owner: 10Hashar) [08:56:49] hashar: seems fine. There is one linting note but otherwise lgtm. Commenting [08:57:44] hashar: btw, some of those arrays are automagically transformed to ferm macros [08:58:26] akosiaris: ah I haven't looked at ferm::rule enhancements :/ [08:58:33] if there is a better way to do it, I am all for it [08:58:47] look in /etc/ferm/conf.d/00_defs [08:59:05] some of the constructs in network.pp are used to populate that file [08:59:26] but not all :-( [08:59:50] I should document it better... [09:00:25] (03CR) 10Alexandros Kosiaris: [C: 04-1] "pedantic note, otherwise LGTM" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148640 (owner: 10Hashar) [09:01:41] akosiaris: I also renamed the ferm rule from gearman_localhost_only to gearman_from_zuul_mergers [09:02:00] if I remember correctly, the ferm conf.d directory is managed by puppet and the old file will be removed [09:02:09] (03CR) 10Hashar: zuul: allow gearman access from merger (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148640 (owner: 10Hashar) [09:02:19] yes it will [09:02:21] (03PS5) 10Hashar: zuul: allow gearman access from mergers [operations/puppet] - 10https://gerrit.wikimedia.org/r/148640 [09:02:23] \O/ [09:03:09] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] zuul: allow gearman access from mergers [operations/puppet] - 10https://gerrit.wikimedia.org/r/148640 (owner: 10Hashar) [09:04:43] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [09:12:56] (03CR) 10Hashar: "Worked fine:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148640 (owner: 10Hashar) [09:14:22] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Puppet has 1 failures [09:15:28] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] "Merging this as discussed." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148386 (owner: 10BryanDavis) [09:16:37] puppet is screwing me :( [09:16:45] we can include zuul several time [09:16:51] but not class { 'zuul': } [09:18:13] hashar, do you know if there is a way to kick jenkins when it doesn't do what it suppose to? [09:18:28] (not right now, just in general - i keep runnig into that problem :( [09:18:33] especially during my depl :) [09:21:33] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 1 failures [09:26:52] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 9 below the confidence bounds [09:27:46] yurikSPBru: what kind of problem? [09:28:02] if jenkins come in your way, it need to be fixed :] [09:28:04] not worked around [09:28:10] :) [09:28:38] it seems every time i try to do a depl, and add the few zero ext to the wmf-release branches [09:28:42] it dies [09:28:47] and i have to override it [09:28:55] any example? [09:29:06] yesterday evening Zuul had some kind of outage that lasted for 2 - 3 hours [09:29:14] the changes were super slow to process [09:30:05] yesterday - it wouldn't pass the rendering testing [09:30:12] see my core depls [09:30:14] (03CR) 10Alexandros Kosiaris: [C: 032] Specify owner/group/mode for apache::conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/148652 (owner: 10Alexandros Kosiaris) [09:32:22] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [09:36:52] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [09:37:38] (03CR) 10Hashar: "lets include role::zuul::install instead of zuul." [operations/puppet] - 10https://gerrit.wikimedia.org/r/141758 (owner: 10Hashar) [09:37:49] (03PS5) 10Hashar: contint: install Zuul on all CI slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/141758 [09:42:02] PROBLEM - check if dhclient is running on virt1008 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:42:13] (03CR) 10Alexandros Kosiaris: [C: 031] "TBH, I started out being negative reviewing this thinking that, hey" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148723 (owner: 10Ori.livneh) [09:43:56] (03CR) 10Hashar: "Puppet compilation: http://puppet-compiler.wmflabs.org/181/change/141758/html/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/141758 (owner: 10Hashar) [09:44:52] RECOVERY - check if dhclient is running on virt1008 is OK: PROCS OK: 0 processes with command name dhclient [09:48:02] PROBLEM - check if dhclient is running on virt1008 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:48:52] RECOVERY - check if dhclient is running on virt1008 is OK: PROCS OK: 0 processes with command name dhclient [09:51:56] (03CR) 10Alexandros Kosiaris: apache::monitoring: add diamond support; ensure mod_status is enabled (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148755 (owner: 10Ori.livneh) [09:55:52] (03PS6) 10Hashar: contint: install Zuul on all CI slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/141758 [09:59:01] (03CR) 10Hashar: [C: 031] "Rebased and recompiled http://puppet-compiler.wmflabs.org/182/change/141758/html/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/141758 (owner: 10Hashar) [09:59:32] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [09:59:45] akosiaris: got another patch that should be fine now https://gerrit.wikimedia.org/r/#/c/141758/ :-D [10:00:06] adds a bunch of includes around to have Zuul source code installed on lanthanum (Jenkins production slave) :D [10:00:13] puppet catalog compiler is happy about it [10:06:54] (03CR) 10Alexandros Kosiaris: [C: 032] contint: install Zuul on all CI slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/141758 (owner: 10Hashar) [10:08:03] akosiaris: thanks :) [10:10:01] !log Zuul code being installed on lanthanum.eqiad.wmnet Will let us use a merger daemon there and the Zuul cloner client. {{gerrit|141758}} [10:10:07] Logged the message, Master [10:10:23] lanthanum:~$ zuul-cloner --version [10:10:23] Zuul version: 2.0.0.200.g57a8557 [10:10:24] awesome [10:11:09] _joe_ akosiaris: Thank you a ton for all the Zuul puppet patches reviews. I finally got Zuul code installed on all Jenkins slaves which is a nice achievement [10:11:14] + cleaner manifests [10:13:09] :-) :-) [10:14:42] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Epic puppet fail [10:14:50] f*** [10:15:02] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet has 1 failures [10:15:13] probably because I aborted puppet run on gallium [10:15:22] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet has 1 failures [10:15:33] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 1 failures [10:15:42] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [10:16:42] PROBLEM - puppet last run on db69 is CRITICAL: CRITICAL: Puppet has 1 failures [10:17:22] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 1 failures [10:18:13] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Puppet has 1 failures [10:23:53] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 5 below the confidence bounds [10:25:33] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [500.0] [10:30:02] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [10:30:12] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [10:30:22] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [10:30:22] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [10:30:32] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [10:31:42] RECOVERY - puppet last run on db69 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [10:38:09] (03CR) 10Alexandros Kosiaris: [C: 032] redis: qualify vars [operations/puppet] - 10https://gerrit.wikimedia.org/r/148621 (owner: 10Matanya) [10:44:01] and it is lunch time [10:44:33] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [10:55:45] (03CR) 10Filippo Giunchedi: [C: 031] "sounds like indeed the overhead is minimal when not enabled" [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/148850 (owner: 10EBernhardson) [11:00:20] (03CR) 10Filippo Giunchedi: [C: 031] "+1 modulo what Alex commented :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148755 (owner: 10Ori.livneh) [11:00:52] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [11:45:32] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 1 failures [11:46:22] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 1 failures [11:46:32] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 1 failures [11:46:33] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 1 failures [11:47:32] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 1 failures [12:00:22] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [12:00:32] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [12:00:32] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [12:00:32] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [12:00:33] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [12:52:37] (03CR) 10Physikerwelt: WIP: Draft for Mathoid role (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 (owner: 10Physikerwelt) [13:13:31] (03PS1) 10Matanya: lucene: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/148992 [13:20:32] (03CR) 10Tim Landscheidt: "Missing newline in ${store}/hostkey-${::fqdn}." [operations/puppet] - 10https://gerrit.wikimedia.org/r/124001 (owner: 10Tim Landscheidt) [13:27:54] (03CR) 10Ottomata: "Hiya," [operations/puppet] - 10https://gerrit.wikimedia.org/r/148847 (owner: 10Yuvipanda) [13:29:05] heya, this is a proposed cname addition [13:29:06] i'm for it [13:29:06] https://gerrit.wikimedia.org/r/#/c/148714/ [13:29:18] but I feel like I should get a +1 from someone else on it [13:31:02] (03CR) 10Yuvipanda: "Heya!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148847 (owner: 10Yuvipanda) [13:31:11] ottomata: ^ [13:32:06] (03PS2) 10Ottomata: statistics: Add packages for rgdal [operations/puppet] - 10https://gerrit.wikimedia.org/r/148847 (owner: 10Yuvipanda) [13:32:43] ottomata: seems simply enough. Not sure I understand qchris's comment though [13:32:49] (03CR) 10Giuseppe Lavagetto: [C: 031] "LGTM but let me first merge the changes I did on the build host :)" [operations/debs/hhvm] - 10https://gerrit.wikimedia.org/r/148850 (owner: 10EBernhardson) [13:33:27] (03PS3) 10Ottomata: statistics: Add packages for rgdal [operations/puppet] - 10https://gerrit.wikimedia.org/r/148847 (owner: 10Yuvipanda) [13:33:54] akosiaris: qchris wants to use those values in a config file, so we don't have to re-do a bunch of stuff (stop and restart long running hadoop/oozie jobs) if we ever decided to change namenode and resourcemanager hosts [13:34:38] akosiaris: it seems harmless enough to me, but I don't know how testy people are about dns changes :), I think its fine too. +1 please? [13:35:16] if there are active connections you will have to stop/restart stuff [13:35:24] but you won't have to reconfigure [13:35:36] (03CR) 10Ottomata: [C: 032 V: 032] statistics: Add packages for rgdal [operations/puppet] - 10https://gerrit.wikimedia.org/r/148847 (owner: 10Yuvipanda) [13:35:45] naw, akosiaris, they aren't active [13:35:46] but ja [13:36:01] oozie launches a new job using that config once every hourish [13:36:13] its the new job that uses the config [13:37:06] but, the config is associated once when the oozie job (über job submitter) is first submitted [13:37:26] (03CR) 10Alexandros Kosiaris: [C: 031] Add aliases for analytics cluster [operations/dns] - 10https://gerrit.wikimedia.org/r/148714 (owner: 10QChris) [13:37:29] danke [13:37:45] (03PS3) 10Ottomata: Add aliases for analytics cluster [operations/dns] - 10https://gerrit.wikimedia.org/r/148714 (owner: 10QChris) [13:37:52] (03CR) 10Ottomata: [C: 032 V: 032] Add aliases for analytics cluster [operations/dns] - 10https://gerrit.wikimedia.org/r/148714 (owner: 10QChris) [13:38:10] !log Deleting old Zuul references in the Zuul maintained repository /srv/ssd/zuul/git/mediawiki/core/ on gallium {{bug|68481}} . Should speed up merge operations on that repository. [13:38:16] Logged the message, Master [13:44:43] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [13:44:52] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [13:45:39] oh, akosiaris, what's the kafka .deb status, should I build the final one and upload to apt? [13:46:45] ottomata: I have been meaning to ask you. http://apt.wikimedia.org/kafka [13:46:56] ready for upload as long as you give a thumbs up [13:47:22] btw I have been reading a bit more on git-buildpackage and I am gonna change the way the branches/tags are to make them more clear [13:47:58] I will document it of course , probably in README.debian [13:48:14] <_joe_> and to ops@ :) [13:48:25] <_joe_> so that everyone will be aware of the best practices [13:49:03] _joe_: yup. Still a draft but ottomata, godog and me are working on it [13:49:17] <_joe_> thanks :) [13:50:56] hashar, are there any rules about deploying minor things now? [13:51:16] yurikSPBru: What is it? And how minor? [13:51:33] There's a SWAT window in just over an hour [13:52:09] Reedy, yeah, but i would rather do it myself if noone else is pushing anything [13:52:11] https://gerrit.wikimedia.org/r/#/c/148948/ [13:52:24] and https://gerrit.wikimedia.org/r/#/c/148907/ [13:52:55] (03PS1) 10Filippo Giunchedi: swift-thumb-stats: dump thumb stats from swift [operations/software] - 10https://gerrit.wikimedia.org/r/148997 [13:56:06] (03CR) 10Qgil: "Could you resolve this old changeset, please? It seems that it was about to be merged. Then again, the repository has been deprecated. Sho" [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [13:56:49] akosiaris: cool, it should be good to go by me. you think we should wait for you gbp changes? probably won't actually change the install result, right? [13:57:27] ottomata: no, they wont. Just cleanups, so feel free to go on [13:57:54] ok cool [13:58:02] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [13:58:06] gage is goign to lead the upgrade, i'll talk to him later today about it [13:58:11] Reedy, so any thoughts? [13:58:25] i'll try not to break it :) [13:59:02] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [13:59:14] yurikSPBru: whatever Reedy says :-] [14:01:11] (03CR) 10Ottomata: "Aye, I'm not sure. It wasn't merged at the time due to some internal discussions about including puppet things in our repos that we weren" [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [14:02:02] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [14:04:02] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [14:04:14] weird... [14:04:37] <_joe_> eww [14:04:46] <_joe_> that looks bad [14:05:08] <_joe_> and I'm searching for an excuse not to submit my patch [14:05:37] yurikSPBru: There's nothing scheduled, so I think you'd be ok. Just add yourself to the deployment calendar anyway for now [14:05:58] Reedy, oki [14:06:50] _joe_: https://gerrit.wikimedia.org/r/#/c/145397/ :) [14:08:01] Reedy, before i start - do you know anything about ttf fonts on our servers? i'm looking for the basic one that supports the largest number of languages (style is not really important) [14:09:12] gah, RT doesn't seem to put anything special in emails for tickets that I have bookmarked or are otherwise owned by me (what I want is to have tickets I'm interested in appear in inbox) [14:10:27] <_joe_> Reedy: ehy sorry about that [14:10:38] <_joe_> btw, luasandbox has no license file [14:11:02] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [14:11:06] It's probably GPL or similar... [14:11:42] yurikSPBru: We install a stack of fonts for SVG and other multimedia usage [14:12:02] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [14:12:59] yurikSPBru: Just use Comic Sans MS [14:13:19] thx! [14:14:20] Reedy, i used /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf for my rendering [14:14:22] hope its ok [14:15:10] Is it definitely installed on the apaches? [14:15:48] Reedy, its on betalabs [14:15:56] checking on tin [14:16:38] it's on mw1017 [14:20:40] (03PS1) 10Matanya: swift: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/149003 [14:21:02] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [14:21:18] <_joe_> is someone taking a look? [14:21:23] <_joe_> if not, I'll do [14:21:33] Reedy, yeah, i see it there too [14:23:02] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [14:24:20] <_joe_> there is probably some sort of bruteforce attack going on [14:24:41] that is exactly what is going on [14:24:52] but it is a DDoS [14:25:06] AFK while I walk the dog [14:25:08] still looking into how to block it, but I think I am gonna do that on cr1,2 [14:26:38] (03CR) 10Yurik: [C: 032] Zero banner image font [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148948 (owner: 10Yurik) [14:26:50] (03Merged) 10jenkins-bot: Zero banner image font [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148948 (owner: 10Yurik) [14:28:22] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [14:30:24] !log yurik Synchronized wmf-config/mobile.php: Font for zero banner (duration: 01m 10s) [14:30:29] Logged the message, Master [14:35:52] (03PS3) 10Giuseppe Lavagetto: Bump apc.shm_size to 360M [operations/puppet] - 10https://gerrit.wikimedia.org/r/145397 (owner: 10Reedy) [14:36:03] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Bump apc.shm_size to 360M [operations/puppet] - 10https://gerrit.wikimedia.org/r/145397 (owner: 10Reedy) [14:36:43] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [14:37:20] <_joe_> ottomata: I merged your changes, FYI [14:37:23] <_joe_> Reedy: done [14:37:43] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [14:39:15] <_joe_> ok so just confirmed - first time this happens to me and I'm using sudo -i [14:39:29] <_joe_> so this is not related to any env variable it seems [14:39:31] oh, oops, thanks _joe_, geez [14:39:34] i'm getting worse at that! [14:40:12] <_joe_> and we have an alert :) [14:40:43] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [14:43:52] hashar, is jenkins having issues again? :( [14:43:56] https://gerrit.wikimedia.org/r/#/c/149010/ [14:44:05] none I know of [14:44:11] paste the error please [14:44:23] hashar, https://gerrit.wikimedia.org/r/#/c/149010/ [14:44:36] hashar, https://gerrit.wikimedia.org/r/#/c/149010/ [14:45:09] (03PS1) 10Hedonil: lighttpd-starter: include default php-fcgi if no .lighttpd.conf is present [operations/puppet] - 10https://gerrit.wikimedia.org/r/149015 (https://bugzilla.wikimedia.org/68499) [14:45:54] hashar, the top 4 - i tried it twice for both 13 & 14 branches, jenkins even succeeded the testing ones, but failed on merge. owner:"Yurik " status:open [14:45:59] https://gerrit.wikimedia.org/r/#/q/owner:%22Yurik+%253Cyurik%2540wikimedia.org%253E%22+status:open,n,z [14:49:06] (03PS4) 10Tim Landscheidt: Tools: Remove lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/124001 [14:49:06] _joe_: awesome, thnaks [14:49:42] (03CR) 10Tim Landscheidt: "Tested on Toolsbeta." [operations/puppet] - 10https://gerrit.wikimedia.org/r/124001 (owner: 10Tim Landscheidt) [14:52:47] yurikSPBru: looking [14:53:08] though my network connection is quite slow :d [14:54:12] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 12:53:16 UTC [14:54:17] yurikSPBru: ah that is interesting so yeah I know the issue :-D [14:54:37] aaand? :) [14:55:02] should i just force +2 it? [14:55:10] or not [14:55:11] hmm [14:55:17] that is only wmf branches [14:55:25] glad you like my problems :) [14:55:49] we need to get rid of those wmf branches eventually :D [14:56:05] sure thing, but i want to deploy something before we do ;) [14:56:53] ValueError: need more than 1 value to unpack [14:56:54] damn git [14:56:59] 2014-07-24 14:56:46,705 ERROR zuul.Merger: Unable to reset repo [14:57:07] probably because I have script running [14:58:10] yurikSPBru: just force merge them [14:58:41] I will retrigger the broken changes in master branch [14:59:21] hashar, go ahead and force 13 & 14 [14:59:32] or i can do it [14:59:40] and i will sync it [14:59:50] or i can do it [15:00:04] Reedy, gi11es: Sir, Please deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140724T1500), the time has come. At your service [15:00:53] Reedy, gi11es, almost done with my patch (or if you want to sync it, that's fine with me too) - there is a problem with jenkins merging wmf branches [15:01:27] Sir :) [15:02:14] (03PS1) 10Filippo Giunchedi: swift: monitor object/container availability [operations/puppet] - 10https://gerrit.wikimedia.org/r/149019 [15:02:46] hashar, i can't submit - that button disapeared [15:03:51] (03PS1) 10Tim Landscheidt: Tools: Install user-requested packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/149020 (https://bugzilla.wikimedia.org/65974) [15:04:24] yurikSPBru: I just submitted https://gerrit.wikimedia.org/r/149010 [15:05:01] Reedy, thx, syncing, please sync one of the 14th ver too [15:05:04] And https://gerrit.wikimedia.org/r/#/c/149013/ [15:05:10] thx [15:05:22] mediawiki/core is blocked fornow [15:08:16] !log yurik Synchronized php-1.24wmf13/extensions/ZeroBanner: (no message) (duration: 01m 11s) [15:08:21] Logged the message, Master [15:10:19] !log yurik Synchronized php-1.24wmf14/extensions/ZeroBanner: (no message) (duration: 01m 07s) [15:10:24] !log Clearing out old Zuul references on operations/puppet.git might cause merge errors [15:10:24] Logged the message, Master [15:10:30] Logged the message, Master [15:11:22] Reedy, gi11es, done with zero. hashar, i will pick your brain about how to do this myself if it happens again [15:12:43] (03PS2) 10Reedy: Fix reference thumbnail settings syntax [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148707 (owner: 10Gilles) [15:12:51] (03CR) 10Reedy: [C: 032] Fix reference thumbnail settings syntax [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148707 (owner: 10Gilles) [15:12:57] (03Merged) 10jenkins-bot: Fix reference thumbnail settings syntax [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148707 (owner: 10Gilles) [15:13:42] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Thu Jul 24 15:13:41 UTC 2014 [15:13:47] !log swapping disk 8 es1001 [15:13:52] Logged the message, Master [15:14:02] !log reedy Synchronized wmf-config/InitialiseSettings-labs.php: Fix reference thumbnail settings syntax (duration: 00m 13s) [15:14:08] Logged the message, Master [15:16:52] PROBLEM - RAID on es1001 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) [15:19:14] (03CR) 10Andrew Bogott: [C: 032] Tools: Install user-requested packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/149020 (https://bugzilla.wikimedia.org/65974) (owner: 10Tim Landscheidt) [15:24:26] Reedy: thanks [15:24:38] beta thumbnailing looks fine [15:25:03] ACKNOWLEDGEMENT - RAID on es1001 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) Sean Pringle RT 7928 [15:35:16] (03PS1) 10BryanDavis: beta: Re-enable luasandbox [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149029 (https://bugzilla.wikimedia.org/68413) [15:35:19] (03CR) 10jenkins-bot: [V: 04-1] beta: Re-enable luasandbox [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149029 (https://bugzilla.wikimedia.org/68413) (owner: 10BryanDavis) [15:36:06] Why Jenkins? [15:37:01] I'd ignore him and force it [15:37:13] (03CR) 10Reedy: "recheck" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149029 (https://bugzilla.wikimedia.org/68413) (owner: 10BryanDavis) [15:37:15] (03CR) 10jenkins-bot: [V: 04-1] beta: Re-enable luasandbox [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149029 (https://bugzilla.wikimedia.org/68413) (owner: 10BryanDavis) [15:38:52] Reedy: All clear on tin for me to pull that? It's a prod no-op [15:38:58] Yup [15:39:08] coolio [15:39:17] !log temporarily stopping puppet on analytics1027 [15:39:22] Logged the message, Master [15:39:34] (03CR) 10BryanDavis: [C: 032 V: 032] "Go home Jenkins, you're drunk." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149029 (https://bugzilla.wikimedia.org/68413) (owner: 10BryanDavis) [15:39:39] (03Merged) 10jenkins-bot: beta: Re-enable luasandbox [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149029 (https://bugzilla.wikimedia.org/68413) (owner: 10BryanDavis) [15:40:39] (03CR) 10coren: [C: 032 V: 032] "LGTM." [operations/puppet] - 10https://gerrit.wikimedia.org/r/149015 (https://bugzilla.wikimedia.org/68499) (owner: 10Hedonil) [15:40:43] !log Fetched c7ae85e to /a/common on tin; prod no-op needed for beta [15:40:48] Logged the message, Master [15:48:42] (03PS1) 10Hoo man: Add client settings needed for the other projects beta feature [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149031 [15:49:52] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [15:49:52] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [15:50:35] (03PS5) 10Chad: Configure Swift-backed elasticsearch backups [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 [15:52:26] (03CR) 10Chad: [C: 031] "Rebased, switched config to use a different user/key per Faidon's request. Just need said user/key (see RT #7418) and we can get this roll" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 (owner: 10Chad) [15:53:11] <^d> godog: Ping [15:54:17] (03PS1) 10BryanDavis: Revert "beta: Re-enable luasandbox" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149033 [15:54:23] ^d: hey! [15:55:00] <^d> godog: Hey, I added you as a reviewer on the swift change for mw-config. What I could really use though is insight on the associated RT for it. [15:55:03] (03CR) 10BryanDavis: [C: 032] Revert "beta: Re-enable luasandbox" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149033 (owner: 10BryanDavis) [15:55:07] (03Merged) 10jenkins-bot: Revert "beta: Re-enable luasandbox" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149033 (owner: 10BryanDavis) [15:55:43] !log Fetched de8022b to /a/common on tin; prod no-op change needed for beta [15:55:49] Logged the message, Master [15:56:00] ^d: sure, I'll take a look [15:56:21] <^d> I wrote down that you were going to help with swift. Making sure I got that right because _joe_'s already on the RT. [15:58:20] ^d: yep that's right :)) [15:58:41] <^d> Okie dokie. So the RT in question is 7418. [15:58:52] <^d> mw-config change is gerrit 130760 [15:59:19] ^d: do you have some numbers on how big the snapshots are? and how they grow? [16:00:06] <^d> They'll grow pretty slow. [16:00:17] <^d> Let me get some numbers on how big. [16:01:54] ^d: cool! I think once we have some figures it'll be easier to understand how much we can get in there [16:02:01] (03CR) 10Hashar: [C: 04-1] "Seem you based your work on Parsoid manifests which are not in a puppet module." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 (owner: 10Physikerwelt) [16:03:08] <^d> godog: Total index size *with* redundancy is 4.6tb. So roughly 1/3 that size. [16:03:18] (03CR) 10Rush: "what hashar said++" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 (owner: 10Physikerwelt) [16:05:20] (03CR) 10Physikerwelt: "Somehow the time where I needed this has passed." [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [16:05:30] <^d> godog: re: the network bottleneck, we can throttle the backup data transfer rate. [16:05:45] ^d: ok! can you put that and how much you'd expect it to grow in the rt so we leave a trail? [16:05:53] <^d> I have it set to 20mb/s but we can lower that if it's a problem. [16:06:41] will it do incremental too? [16:07:04] I mean, not 1tb at 20mb/s at each snapshot [16:07:10] <^d> It's incremental. [16:07:14] <^d> Only the first dump is full. [16:09:43] (03PS1) 10Reedy: Swap wikimania2013wiki to wikimania2014wiki in wmgCentralAuthLoginIcon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149036 [16:09:59] (03CR) 10Reedy: [C: 032] Swap wikimania2013wiki to wikimania2014wiki in wmgCentralAuthLoginIcon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149036 (owner: 10Reedy) [16:10:03] (03Merged) 10jenkins-bot: Swap wikimania2013wiki to wikimania2014wiki in wmgCentralAuthLoginIcon [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149036 (owner: 10Reedy) [16:10:25] !log reedy Synchronized wmf-config/InitialiseSettings.php: Swap wikimania2013wiki to wikimania2014wiki in wmgCentralAuthLoginIcon (duration: 00m 14s) [16:10:31] Logged the message, Master [16:13:46] * Nemo_bis is currently tarballing Commons [16:14:03] <^d> godog: RT updated with my napkin math :) [16:14:24] Nemo_bis: as in the text? good luck with the images [16:17:20] no, images (downloading from your.org where available) [16:18:04] it'll be in the order of tens of TBs, fwiw :) [16:18:07] I already did the earliest 24 TB last year, now doing the remaining 10 TB of 2013 https://archive.org/details/wikimediacommons [16:18:41] hah, nice [16:31:05] (03CR) 10Filippo Giunchedi: Configure Swift-backed elasticsearch backups (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 (owner: 10Chad) [16:34:12] (03CR) 10BryanDavis: WIP: Draft for Mathoid role (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 (owner: 10Physikerwelt) [16:34:32] (03PS6) 10Chad: Configure Swift-backed elasticsearch backups [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 [16:35:05] (03CR) 10Chad: "PS6 lowers throttle to 10mb/s per request, also uses correct variable name (which seems to have changed on me since I wrote this originall" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 (owner: 10Chad) [16:37:35] (03CR) 10Filippo Giunchedi: [C: 031] "sounds good, will create and communicate the account data and try it starting next week" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 (owner: 10Chad) [16:37:59] !log restarting Jenkins it is broken again [16:38:04] Logged the message, Master [16:55:05] <^d> ottomata: Fun fact: we don't even consistently use noatime. [16:55:11] <^d> elastic1001 has it. [16:55:13] <^d> 1012 doesn't. [16:55:22] <^d> 1017-1019 don't, obviously. [16:55:25] <^d> Didn't check the rest. [17:12:51] (03CR) 10Mwalker: "Without looking, yet, at the patch -- following up on hashar -- you can look at the PDF rendering module and role (it's called OCG)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 (owner: 10Physikerwelt) [17:15:14] (03PS1) 10BBlack: fix mmap-fail printf formatting [operations/debs/varnish] (3.0.6-plus-wm) - 10https://gerrit.wikimedia.org/r/149049 [17:15:16] (03PS1) 10BBlack: Added new patch for explicit persistent mmap addr [operations/debs/varnish] (3.0.6-plus-wm) - 10https://gerrit.wikimedia.org/r/149050 [17:15:18] (03PS1) 10BBlack: varnish (3.0.6plus~x-wm2) unstable; urgency=low [operations/debs/varnish] (3.0.6-plus-wm) - 10https://gerrit.wikimedia.org/r/149051 [17:15:57] (03CR) 10BBlack: [C: 032] fix mmap-fail printf formatting [operations/debs/varnish] (3.0.6-plus-wm) - 10https://gerrit.wikimedia.org/r/149049 (owner: 10BBlack) [17:16:07] (03CR) 10BBlack: [C: 032] Added new patch for explicit persistent mmap addr [operations/debs/varnish] (3.0.6-plus-wm) - 10https://gerrit.wikimedia.org/r/149050 (owner: 10BBlack) [17:16:18] (03CR) 10BBlack: [C: 032] varnish (3.0.6plus~x-wm2) unstable; urgency=low [operations/debs/varnish] (3.0.6-plus-wm) - 10https://gerrit.wikimedia.org/r/149051 (owner: 10BBlack) [17:17:09] (03PS1) 10Reedy: labs max lag to 300 seconds [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149052 [17:19:21] (03PS2) 10Ori.livneh: apache::monitoring: add diamond support; ensure mod_status is enabled [operations/puppet] - 10https://gerrit.wikimedia.org/r/148755 [17:19:45] (03CR) 10Ori.livneh: [C: 032 V: 032] apache::monitoring: add diamond support; ensure mod_status is enabled [operations/puppet] - 10https://gerrit.wikimedia.org/r/148755 (owner: 10Ori.livneh) [17:20:13] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:19:46 UTC [17:20:55] (03PS1) 10Mwalker: Give OCG admins ability to restart the service [operations/puppet] - 10https://gerrit.wikimedia.org/r/149053 [17:21:21] Coren: ok to merge "Hedonil: lighttpd-starter: include default php-fcgi if no .lighttpd.conf is present (8354066253)"? [17:21:55] looks safe, /me does. [17:22:52] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [17:22:53] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [17:24:17] (03PS1) 10Reedy: Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149055 [17:24:33] (03CR) 10Reedy: [C: 032] Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149055 (owner: 10Reedy) [17:25:30] (03Merged) 10jenkins-bot: Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149055 (owner: 10Reedy) [17:25:51] (03PS2) 10Reedy: labs max lag to 300 seconds [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149052 [17:26:00] (03CR) 10Reedy: [C: 032] labs max lag to 300 seconds [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149052 (owner: 10Reedy) [17:26:04] (03Merged) 10jenkins-bot: labs max lag to 300 seconds [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149052 (owner: 10Reedy) [17:26:51] chrismcmahon: ^ [17:26:51] see if that helps... [17:27:12] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: Puppet has 1 failures [17:28:56] (03CR) 10Jgreen: [C: 032 V: 031] Give OCG admins ability to restart the service [operations/puppet] - 10https://gerrit.wikimedia.org/r/149053 (owner: 10Mwalker) [17:30:32] thanks Reedy [17:30:43] (03PS1) 10Reedy: testwiki to 1.24wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149056 [17:30:45] (03PS1) 10Reedy: Wikipedias to 1.24wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149057 [17:30:47] (03PS1) 10Reedy: group0 to 1.24wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149058 [17:31:05] (03PS1) 10Mwalker: Add OCG servers to the PDF ganglia cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/149059 [17:31:13] (03CR) 10Reedy: [C: 032] testwiki to 1.24wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149056 (owner: 10Reedy) [17:31:21] (03Merged) 10jenkins-bot: testwiki to 1.24wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149056 (owner: 10Reedy) [17:31:32] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Puppet has 1 failures [17:32:42] !log reedy Started scap: testwiki to 1.24wmf15 and build l10n cache [17:32:43] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Puppet has 1 failures [17:32:48] Logged the message, Master [17:33:42] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 1 failures [17:33:43] PROBLEM - puppet last run on ssl1008 is CRITICAL: CRITICAL: Puppet has 1 failures [17:34:12] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [17:34:56] (03CR) 10Jgreen: [C: 032 V: 031] Add OCG servers to the PDF ganglia cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/149059 (owner: 10Mwalker) [17:35:43] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Puppet has 1 failures [17:36:22] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 1 failures [17:36:42] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 1 failures [17:39:42] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 1 failures [17:39:42] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: Puppet has 1 failures [17:41:22] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 1 failures [17:44:42] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 1 failures [17:46:21] (03PS1) 10Ori.livneh: apache: cherry-pick mods added in Ia46312071 [operations/puppet] - 10https://gerrit.wikimedia.org/r/149066 [17:48:19] (03CR) 10Ori.livneh: [C: 032] admin: fix my bash aliases [operations/puppet] - 10https://gerrit.wikimedia.org/r/148876 (owner: 10Ori.livneh) [17:49:14] something broke apache config on the ssl terminators :P [17:49:37] bblack: could be me, i've been doing work on that stack. got a hostname handy? [17:49:50] any of the ones icinga is spamming about above? [17:50:02] oops, i muted puppet freshness alerts yesterday [17:50:26] ok got it [17:50:32] that seems unwise given the odds of your causing them :P [17:51:57] I wonder if it could be done in batches at least, that seems more trouble than it is worth tho [17:51:58] yes [17:52:05] are they supposed to have apache? [17:52:12] the most recent puppet run appears to have installed it [17:52:38] I doubt they're supposed to have it, but I'm not 100% sure any more than you are [17:52:59] they run nginx [17:53:12] somehow my change caused them to provision apache, which failed to start because nginx is already listening on the relevant ports [17:53:17] now to find out why [17:53:44] (03PS1) 10BBlack: Add explicit mmap addrs for varnish persistent storage [operations/puppet] - 10https://gerrit.wikimedia.org/r/149068 [17:54:10] modules/protoproxy/manifests/ganglia.pp:5: include ::apache::monitoring [17:54:15] why would it do that? [17:54:17] Reedy: how's the deploy today? anything crazy/dumb going on right now? [17:54:33] i'm crazy and dumb [17:54:36] (03CR) 10BBlack: [C: 04-1] "Note we won't have the relevant package (3.0.6-plus~x-wm2 or higher) until upstream finishes releasing 3.0.6 and we merge that, etc. So t" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149068 (owner: 10BBlack) [17:55:16] ok, i think i know what this is about [17:55:32] protoproxy kept the apache::monitoring include probably so that the data isn't purged from ganglia [17:55:43] ori: serious or no? :) [17:55:48] not serious [17:55:51] whew [17:56:02] when we migrated from apache -> nginx [17:56:07] anyways, it has no place there now [17:56:14] are you sure? [17:56:20] yes [17:56:28] the graphs in ganglia are blank [17:56:36] ok [17:56:43] going as far back as a year [17:57:03] greg-g: scap-rebuild-cdbs: 81% (ok: 184; fail: 0; left: 44) [17:57:13] ah [17:57:36] so sloooooow [18:00:02] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Thu Jul 24 18:00:00 UTC 2014 [18:00:05] Reedy, greg-g: Sir, Please deploy MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140724T1800), the time has come. At your service [18:00:16] scap-rebuild-cdbs: 100% (ok: 227; fail: 0; left: 1) [18:00:19] * Reedy waits [18:00:53] (03PS1) 10Ori.livneh: protoproxy: don't include apache::monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/149069 [18:01:04] (03Abandoned) 10Spage: Revert "Move Mantle dependency to MobileFrontend too, given Ia060b2295" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148931 (owner: 10Spage) [18:02:43] (03CR) 10Ori.livneh: "Doesn't appear to be migration-related; it was never sane for this to be part of the protoproxy module. See change d0b881e68c574b6d25743e1" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149069 (owner: 10Ori.livneh) [18:02:43] stuff to undo on ssl's: http://paste.debian.net/hidden/23347256/ [18:02:57] yes, i was just gearing up for that [18:02:59] thanks [18:03:05] could you sanity-check the patch? [18:03:22] we can salt some undo commands instead of trying to do it via puppet [18:03:31] yes of course [18:03:37] the patch just stops applying apache::monitoring [18:03:46] i wasn't going to ensure => absent all the pieces [18:03:49] i'll !log as i clean up [18:03:55] !log reedy Finished scap: testwiki to 1.24wmf15 and build l10n cache (duration: 31m 12s) [18:03:56] (03CR) 10BBlack: [C: 031] protoproxy: don't include apache::monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/149069 (owner: 10Ori.livneh) [18:04:00] Logged the message, Master [18:04:13] (03CR) 10Ori.livneh: [C: 032] protoproxy: don't include apache::monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/149069 (owner: 10Ori.livneh) [18:06:49] my take on the puppet log is basically 3 actions to undo: kill /etc/diamond/collectors/HttpdCollector.conf , remove apache2 + apache2-mpm-worker pkgs, rm -rf /etc/apache2 [18:07:02] well and refresh diamond [18:07:20] diamond collectors are recursively managed so the patch caused them to be removed and the service refreshed, which is good. [18:07:28] Info: /Stage[main]/Diamond/File[/etc/diamond/collectors/HttpdCollector.conf]: Filebucketed /etc/diamond/collectors/HttpdCollector.conf to puppet with sum 65a8d826eb05ea086f783f61e33a78ca [18:07:28] Notice: /Stage[main]/Diamond/File[/etc/diamond/collectors/HttpdCollector.conf]/ensure: removed [18:07:30] ah that's nice [18:07:43] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:08:14] i'll do apt-get remove --dry-run [18:08:56] The following packages were automatically installed and are no longer required: [18:08:57] libaprutil1-ldap apache2-mpm-prefork libruby apache2-utils apache2.2-common libaprutil1-dbd-sqlite3 apache2.2-bin libzmq3 libaprutil1 [18:09:07] some of these are not related so i don't want to do an autoremove [18:09:15] ? [18:09:31] e.g. i don't see how libzmq3 is obsoleted by removing apache [18:09:32] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:09:43] yeah maybe [18:09:47] $ aptitude why libzmq3 [18:09:47] Unable to find a reason to install libzmq3. [18:09:55] probably an expierment [18:09:58] what host was that? [18:10:03] ssl1003 [18:10:38] 'aptitude why libaprutil1-ldap' shows that that package _was_ installed for apache [18:10:43] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:10:59] libruby is unrelated [18:11:33] well [18:11:42] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:11:43] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:11:50] we could at least explicitly hit the obvious ones like apache2-common and apache2.2-bin [18:12:10] yeah, could you look at http://paste.debian.net/hidden/cff981ba/ ? [18:12:30] Reedy: Did _joe_ do apache restarts to go with https://gerrit.wikimedia.org/r/#/c/145397/? [18:12:42] I'm not sure [18:12:43] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:12:49] I don't recall seeing any [18:13:01] bd808|LUNCH: check when the process was started i guess [18:13:12] Yeah graphs don't seemed to have changed upwards. [18:13:56] ori: that looks good to me. -G cluster:ssl target on palladium I assume [18:14:22] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:14:42] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [18:14:51] /var/log/dpkg.log confirms that those are the packages that were installed [18:15:01] (03PS1) 10Spage: Correct variable for MobileFrontend Mantle dep [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149071 [18:15:03] i'll go ahead in 1 min unless someone screams murder [18:15:34] actually i'll start with just ssl1003 [18:15:41] then the rest of the ssl cluster via salt [18:15:46] (03CR) 10Reedy: [C: 032] Wikipedias to 1.24wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149057 (owner: 10Reedy) [18:15:50] (03Merged) 10jenkins-bot: Wikipedias to 1.24wmf14 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149057 (owner: 10Reedy) [18:16:26] sounds good [18:16:42] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:17:28] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf14 [18:17:34] Logged the message, Master [18:18:09] looks good on ssl1003. forced a puppet run after removing and confirmed it's a no-op. so doing the rest now. [18:18:22] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:21:28] !log updated swift ring to bring ms-be1013 weight to 2300 [18:21:33] Logged the message, Master [18:21:33] (03CR) 10Reedy: [C: 032] group0 to 1.24wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149058 (owner: 10Reedy) [18:21:38] (03Merged) 10jenkins-bot: group0 to 1.24wmf15 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149058 (owner: 10Reedy) [18:21:42] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:21:47] (03PS1) 10EBernhardson: $wmgUseMobileFrontend does not exist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149073 [18:22:39] (03PS1) 10Jdlrobson: Use correct variable in config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149074 [18:23:03] ori: legoktm Krinkle - Lydia_WMDE - would be great if you could reply to http://lists.wikimedia.org/pipermail/wikitech-l/2014-July/077554.html - thanks! [18:23:09] !log Purged apache from SSL cluster; provisioned as a side-effect of I0b02a46f3 + I76a0d237f [18:23:13] Logged the message, Master [18:23:30] bblack: everything looks good. thanks for the help. [18:23:32] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 1 failures [18:23:44] looking [18:24:12] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:24:15] sumanah: The RCStream wiki page documents the implemented code. The code has been implemented, moved into gerrit, reviewed by peers, approved by ops, and is already live in production. It hasn't been publicly launched widely, but you can consider it a public beta. [18:24:15] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf15 [18:24:21] Logged the message, Master [18:24:22] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 1 failures [18:24:33] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 1 failures [18:25:00] sigh, fixed permissions on the ring files [18:25:27] understood Krinkle [18:25:27] (03PS2) 10Reedy: Correct variable for MobileFrontend Mantle dep [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149071 (owner: 10Spage) [18:25:42] (03CR) 10Reedy: [C: 032] Correct variable for MobileFrontend Mantle dep [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149071 (owner: 10Spage) [18:25:46] (03Merged) 10jenkins-bot: Correct variable for MobileFrontend Mantle dep [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149071 (owner: 10Spage) [18:26:29] sumanah: No big announcements to the world at large, initially reaching out to internal infrastructure and developers whom we know are currently using the legacy irc.wikimedia.org interface so that they can migrate first, and so we can see whether it suites their usecases. Basically while it is stable and running, we might still change it any time without keeping the current version compatible. [18:26:29] Quick iteration. [18:26:35] !log reedy Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 14s) [18:26:40] Logged the message, Master [18:26:43] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 1 failures [18:26:51] (03Abandoned) 10Reedy: $wmgUseMobileFrontend does not exist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149073 (owner: 10EBernhardson) [18:26:52] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 1 failures [18:26:56] (03Abandoned) 10Reedy: Use correct variable in config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149074 (owner: 10Jdlrobson) [18:26:59] ori: Can you confirm? [18:27:22] (my prev 3 messages in this channel) [18:27:30] * ori looks [18:28:27] yes [18:28:48] (03CR) 10Reedy: "This should be good to go now?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/146951 (owner: 10Gergő Tisza) [18:29:11] (03PS1) 10EBernhardson: Enable sandbox page for wikimania user testing [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149077 [18:29:26] (03PS2) 10Reedy: Remove duplicate $wgFlowParsoidURL setting. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148918 (owner: 10Spage) [18:29:34] (03CR) 10Reedy: [C: 032] Remove duplicate $wgFlowParsoidURL setting. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148918 (owner: 10Spage) [18:29:37] (03Merged) 10jenkins-bot: Remove duplicate $wgFlowParsoidURL setting. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148918 (owner: 10Spage) [18:29:48] tgr: gi11es: to access deployment-*, you can ask project membership for the deployment-prep labs project on the qa or ops channel [18:29:50] and so I did [18:31:13] (03PS2) 10Reedy: Remove MMV on private wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145612 (owner: 10MarkTraceur) [18:31:25] (03CR) 10Reedy: [C: 032] Remove MMV on private wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145612 (owner: 10MarkTraceur) [18:31:29] (03Merged) 10jenkins-bot: Remove MMV on private wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145612 (owner: 10MarkTraceur) [18:32:07] (03PS2) 10Reedy: Add lawiki featured feed settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145760 (https://bugzilla.wikimedia.org/33978) (owner: 10TTO) [18:32:11] (03CR) 10Reedy: [C: 032] Add lawiki featured feed settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145760 (https://bugzilla.wikimedia.org/33978) (owner: 10TTO) [18:32:55] (03CR) 10Reedy: [C: 04-1] "[19:32:22] Reedy: think we should wait until tuesday or talk to hoo" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149031 (owner: 10Hoo man) [18:33:22] (03PS3) 10Reedy: Namespaces configuration for ru.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145556 (https://bugzilla.wikimedia.org/67844) (owner: 10Dereckson) [18:33:23] (03CR) 10Aude: "looks okay to me, though assume we wait until tuesday when we have a deploy slot" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149031 (owner: 10Hoo man) [18:33:32] (03CR) 10Reedy: [C: 032] Namespaces configuration for ru.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145556 (https://bugzilla.wikimedia.org/67844) (owner: 10Dereckson) [18:33:37] (03Merged) 10jenkins-bot: Namespaces configuration for ru.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145556 (https://bugzilla.wikimedia.org/67844) (owner: 10Dereckson) [18:34:28] (03PS3) 10Reedy: Set otherProjectsLinksByDefault for kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148719 (owner: 10Hoo man) [18:34:32] (03CR) 10Reedy: [C: 032] Set otherProjectsLinksByDefault for kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148719 (owner: 10Hoo man) [18:34:36] (03Merged) 10jenkins-bot: Set otherProjectsLinksByDefault for kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148719 (owner: 10Hoo man) [18:34:38] !log aaron Synchronized php-1.24wmf14/includes/jobqueue/aggregator/JobQueueAggregatorRedis.php: ca031131396ee1830e239d0b6a314bb571840c11 (duration: 00m 06s) [18:34:43] Logged the message, Master [18:36:35] (03PS3) 10Reedy: Add lawiki featured feed settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/145760 (https://bugzilla.wikimedia.org/33978) (owner: 10TTO) [18:39:22] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 5 unmerged changes in mediawiki_config (dir /a/common/). [18:39:43] !log reedy Synchronized wmf-config/: (no message) (duration: 00m 14s) [18:39:48] Logged the message, Master [18:40:22] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [18:41:32] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [18:41:58] !log Updated jobrunners to 01c70b1a892ac3944655f84449e89e4508894101 [18:42:04] Logged the message, Master [18:42:22] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:42:32] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:43:43] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:43:52] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:44:16] !log restarted jobrunners for 01c70b1a892ac3944655f84449e89e4508894101 [18:44:17] ^ AaronSchulz [18:44:20] Logged the message, Master [18:44:36] nice [18:54:11] (03CR) 10Dzahn: [C: 032] ircecho: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/148347 (owner: 10Matanya) [18:54:55] (03CR) 10Hashar: "We will want to update the varnish on beta cluster as well :) Good to know you found a way to fix the spurious restart failure!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149068 (owner: 10BBlack) [19:03:45] (03CR) 10Dzahn: [C: 031] memcached: qualify vars [operations/puppet] - 10https://gerrit.wikimedia.org/r/148348 (owner: 10Matanya) [19:03:59] Reedy: We may need to revert the train. [19:04:13] greg-g, Reedy: We probably need to revert today's deployment train [19:04:17] See https://en.wikipedia.org/wiki/User:RYasmeen_%28WMF%29/sandbox [19:04:25] Reedy: Finding major wmf13 corr… yes, what RoanKattouw said. [19:04:34] If you're logged out, or logged in as me (but not if you're logged in as James!), you get UNIQ---QINU garbage [19:04:40] But not in preview :S [19:04:58] Also if you edit the page and save without making changes, it replaces the tags with UNIQ garbage in the wikitext [19:05:08] I don't see UNIQ [19:05:14] Reedy: Try viewing as an anon [19:05:26] I do in my personal and anon, but not staff account. [19:05:51] i see the same UNIQ stuff when i am logged in. [19:05:55] I do as anon [19:06:31] bblack: ping [19:07:27] bblack: we need help to deploy https://gerrit.wikimedia.org/r/#/c/148924/ to stop occasional 503s from the parsoid varnishes [19:09:37] Reedy: You decided whether we should revert the train? [19:11:37] How widespread actually is the damage? [19:13:15] Not all pages seem to be affected [19:13:27] http://en.wikipedia.org/wiki/List_of_battles_and_other_violent_events_by_death_toll for instance was recently edited and has references, but is not affected [19:14:02] Ooooh hold on [19:14:08] There's something weird about that user page [19:14:14] There are refs in two groups, but only one group has a reflist [19:14:21] So maybe this bug is in the auto-reflist code? [19:14:54] Yeah it seems to be [19:15:00] Adding to that page fixes it [19:15:27] And removing it breaks it again [19:17:46] Should we pin back Cite to wmf12 then? [19:17:55] ^^^ Reedy / RoanKattouw. [19:18:07] wmf12? [19:18:09] Possibly [19:18:11] you mean 13? [19:18:16] Reedy: This is happening on enwiki [19:18:21] Which is running wmf13, rihgt? [19:18:21] Reedy: No, it's running wmf13 right now, that's the problem. [19:18:33] Oh, no. [19:18:35] Off by one error. [19:18:38] Yeah, 13. [19:18:39] No it's not [19:18:44] * James_F was wrong. [19:18:45] It's running 14 [19:18:46] :P [19:19:19] https://github.com/wikimedia/mediawiki-extensions-Cite/commits/master [19:19:53] We should just revert https://github.com/wikimedia/mediawiki-extensions-Cite/commit/798b4537712ea448b863eaf57800dfec685d6ff7 out of master as broken, and backport to wmf14 and wmf15? [19:20:02] Yeah, https://gerrit.wikimedia.org/r/#/c/145474/ is the only new patch. [19:20:34] But adding a tracking category causing Parser to fail? [19:20:40] It feels… surprising. [19:22:13] Cite is… surprising. [19:22:22] Touché. [19:22:27] https://www.mediawiki.org/wiki/MediaWiki_1.24/wmf14/Changelog [19:22:30] That suggests there's no cite changes in wmf14 [19:22:36] (03CR) 10Hashar: "recheck" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 (owner: 10Chad) [19:22:53] Reedy: Sounds like a bug in the script – https://gerrit.wikimedia.org/r/#/c/145474/ "Included in" says wmf14+. [19:23:31] Confirmed by running git log origin/wmf/1.24wmf13..origin/wmf/1.24wmf14 --oneline [19:23:37] 44472de Provide way for users to find articles with the automatic [19:23:40] That too. [19:23:42] Cite doesn't look to be in any of the logs... [19:23:50] is the only thing that's not a localization update or "Creating new wmf/1.24wmf14 branch" [19:23:53] Maybe it has so few changes we never noticed it wasn't listed? [19:24:07] !log restarted Zuul [19:24:12] Logged the message, Master [19:31:32] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.001 second response time [19:31:39] Nikerabbit: http://pastebin.com/JL0JNp5R I thought that used the job queue [19:33:09] heh, looks like the log entry was cut off too...packet too big ;) [19:33:15] Reedy: You OK to deploy the cherry-picks? [19:33:29] AaronSchulz: usually it does, but not always [19:33:32] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.011 second response time [19:33:41] Reedy: https://gerrit.wikimedia.org/r/149096 for wmf14 and https://gerrit.wikimedia.org/r/149093 for wmf15. [19:35:12] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [19:36:52] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 1 failures [19:39:26] James_F, although do we know that this bug wasn't there before tha patch? [19:40:01] subbu: We didn't see it before wmf14, and that's the most likely change. If that revert doesn't work we'll do something else.. [19:40:10] AaronSchulz: is it a problem? I thought it just does nothing when not running via jobqueue? [19:40:43] there most have been some slow method in there...though it got cut off [19:41:00] that's a lot of locks to hold, do stat table updates really need to be synchronous? [19:41:06] k [19:42:53] Blargh, internt connection [19:43:12] cmjohnson1: we need root help to deploy https://gerrit.wikimedia.org/r/#/c/148924/ to stop occasional 503s from the Parsoid varnishes [19:43:43] k [19:44:14] (03CR) 10Cmjohnson: [C: 032] Bump Parsoid backend connection limit [operations/puppet] - 10https://gerrit.wikimedia.org/r/148924 (https://bugzilla.wikimedia.org/68276) (owner: 10GWicke) [19:44:42] cmjohnson1: thx! [19:44:45] gwicke: merged [19:44:47] yw [19:45:04] I'm not 100% sure if those varnishes are implicitly restarted by puppet on config change [19:45:59] let me check after the next puppet run; if they were not restarted I might ask you to restart them as well [19:46:14] okay [19:46:21] /cc RoanKattouw [19:46:54] AaronSchulz: no they most likely don't need to be synchronous (as long as the current process has local cache) [19:50:20] gwicke: I'll restart if needed. Can you do netstat monitoring to check if #conns goes >1000 after the change? [19:53:07] James_F, RoanKattouw: So what's going on with this Cite stuff? [19:53:52] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [19:54:14] Krenair: it's getting reverted, no? [19:54:15] Krenair: https://en.wikipedia.org/wiki/User:RYasmeen_(WMF)/sandbox [19:54:27] Vogone, ... yes that's why I'm asking [19:54:31] note the UNIQs [19:54:33] k [19:54:42] what uniqs? [19:54:52] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [19:54:54] oh yeah, it's apparently not visible for everyone [19:54:56] try logged out [19:55:14] Reedy: RoanKattouw you ok? [19:55:17] wat. [19:55:22] And it's the tracking category causing this? [19:55:34] http://i.imgur.com/h5gbynx.png [19:55:37] Krenair: Basically if you have s and s on a page, and but not , you get strip markers everywhere [19:55:38] apparently [19:55:45] If you add , the strip markers go away [19:55:58] Nikerabbit: can it be changed then? [19:55:59] But the strip markers are only visible for anons and some but not all users [19:56:01] i wonder why not everyone sees it, maybe it depends on some user prefs? [19:56:09] I wonder if this bug is really in Cite and not the parser [19:56:20] Krenair: And yes it appears to be the category change, because that is the only thing that changed between the working version and the broken version [19:56:53] MatmaRex: user language? what about system messages [19:57:04] Oooh [19:57:06] user language could be [19:57:13] James_F said it worked for him [19:57:16] And he probably uses en-gb [19:57:21] I'm also on en-gb [19:57:40] i'm on en [19:58:00] AaronSchulz: if you make a patch I can test it on translatewiki.net for example [19:58:04] (03PS1) 10Jgreen: move fundraising.wm.o off of aluminium and onto a frack host [operations/dns] - 10https://gerrit.wikimedia.org/r/149103 [19:58:35] !log reedy Synchronized php-1.24wmf14: (no message) (duration: 01m 11s) [19:58:41] Logged the message, Master [19:58:42] Nikerabbit: heh, I was hoping some maintainer could do that [19:58:51] yup, has to do with language [19:58:53] * AaronSchulz needs to do more job queue / hhvm stuff [19:58:58] (03CR) 10Jgreen: [C: 032 V: 031] move fundraising.wm.o off of aluminium and onto a frack host [operations/dns] - 10https://gerrit.wikimedia.org/r/149103 (owner: 10Jgreen) [19:59:59] (03PS1) 10Hashar: contint: breaking breaking [operations/puppet] - 10https://gerrit.wikimedia.org/r/149105 [20:00:17] AaronSchulz: I understand, but given I am on vacation I'm putting work-y stuff at the end of the queue so it will take a while [20:00:23] !log reedy Synchronized php-1.24wmf15: (no message) (duration: 00m 59s) [20:00:28] Logged the message, Master [20:01:35] gwicke: yeah varnish doesn't restart on config change, it has too much impact [20:01:53] (and also, it potentially doesn't start again due to bugs, takes multiple manual tries) [20:03:01] AaronSchulz: if you say it's high priority we can of course relay that to LangEng team to handle [20:03:20] it can go in some backlog [20:05:00] AaronSchulz: can you or me file a bug report then? [20:05:09] already did [20:05:24] AaronSchulz: thanks! [20:05:58] (03PS2) 10Hashar: contint: setup localhost.qunit vhost on lanthanum [operations/puppet] - 10https://gerrit.wikimedia.org/r/149105 (https://bugzilla.wikimedia.org/68529) [20:08:20] Reedy: FWIW it now seems to work. [20:08:28] /cc Krenair [20:08:35] lol [20:08:42] Awesome bug is awesome [20:08:46] !log reedy Synchronized php-1.24wmf15/extensions/Translate: (no message) (duration: 00m 15s) [20:08:51] Logged the message, Master [20:11:21] (03CR) 10Hashar: "puppet catalog compilation: http://puppet-compiler.wmflabs.org/183/change/149105/html/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149105 (https://bugzilla.wikimedia.org/68529) (owner: 10Hashar) [20:11:25] any kind ops could merge me in a contint puppet change please? I need a vhost applied on a server https://gerrit.wikimedia.org/r/#/c/149105/ :-D [20:11:32] puppet catalog compiler is happy about it [20:13:24] bblack: okay, thanks for the info! [20:13:41] I'm looking at the change now, something's odd about it [20:14:00] on one of the two parsoid varnishes, it applied then unapplied in successive runs of puppet? the other never saw it [20:14:21] (03CR) 10Spage: [C: 04-1] "Underscores don't work, only spaces." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149077 (owner: 10EBernhardson) [20:14:56] bblack: sounds odd [20:14:57] it's strontium out of sync on puppet-merge [20:15:52] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [20:18:45] MatmaRex, RoanKattouw: Why doesn't the parser throw exceptions when it fails to handle the markers? [20:19:03] lol [20:19:46] that probably isn't necessary, usually [20:20:03] and you can get UNIQ markers to appear in the output in a variety of ways [20:20:22] and the parser is supposed to produce some output for any input, and never "fail" [20:20:24] so i guess that's why [20:20:27] !log restarted backend varnish for parsoid on cp1045 [20:20:32] Logged the message, Master [20:20:39] a better question is, why does it fail to handle the markers in this case? [20:21:12] !log restarted backend varnish for parsoid on cp1058 [20:21:17] Logged the message, Master [20:21:35] bawolff did some work on the parser choking when you nest the calls, or something, maybe he'd be able to figure this out relatively quickly [20:21:48] i think Cite does some very bad things to Parser :( [20:22:58] bblack: thanks! [20:24:42] bblack: on cp1058, the # of backend connections is already op to 1375 this quickly after the restart [20:24:44] MatmaRex, urgh. Why can't it at least log errors somewhere? [20:24:55] bblack: so 1000 was definitely too low [20:25:24] Krenair: that's what tracking categories are for :D [20:25:43] Krenair: kidding, i guess you mean detecting when the UNIQ markers are not processed? [20:26:02] * Krenair grumbles [20:26:25] MatmaRex, yes [20:26:36] i'm not sure if you can detect leftover UNIQ markers if you've lost the data about them (somehow) [20:26:41] well, other than regexing things a lot [20:27:20] Surely it can keep track of markers it left and then forget them when it successfully handles then? [20:27:34] handles them* [20:27:44] Reedy: any follow up re the revert I should know about? I haven't read backscroll completely. [20:28:06] anything I can help with (ensuring things happen etc), that is [20:28:10] bblack: I saw pretty low max_connections settings on other varnish instances like 'misc', do you think we should revisit those? [20:28:37] gwicke: I'm not sure. [20:28:46] Krenair: this bug look like it has *lost* track of them to me [20:28:52] I imagine it's a highly-variable thing depending on how the backends behave [20:28:55] (of course i have nothing to back this wild guess with) [20:29:19] (and we'd much rather have varnish cap out max_conns than actually overload the backends) [20:29:24] (but, when UNIQ markers break, i think it's usually this or somebody regexing too hard over half-parsed content) [20:29:34] but "1000" is pretty arbitrary-sounding, so I doubt anyone really tuned it [20:29:46] bblack: I see.. just wonder if # of connections is a good proxy for load [20:29:56] misc only allows 100 backend connections [20:30:04] probably not always, but it's still good to cap it [20:31:40] which might not be much across the 10+ services it serves [20:32:23] gwicke: http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Misc%20Web%20caching%20cluster%20eqiad&h=cp1043.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1406233897&event=show&ts=0&v=5940&m=varnish.backend_busy&vl=N%2Fs&ti=Backend%20conn.%20too%20many&z=large [20:32:40] ganglia varnish stats can tell us ^ [20:32:56] seems like we had a spike a few days back on misc, but otherwise nothing [20:33:21] k [20:34:43] bblack: strangely there's nothing in ganglia for cp1045: https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Parsoid%20Varnish%20eqiad&h=cp1045.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1406234031&v=0&m=frontend.backend_busy&vl=N%2Fs&ti=Backend%20conn.%20too%20many&z=large [20:35:47] ah, frontend [20:35:48] (03PS1) 10Ottomata: Set authorization_service_authorization_enabled for Oozie. [operations/puppet] - 10https://gerrit.wikimedia.org/r/149112 [20:35:48] nm [20:37:13] https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Parsoid%20Varnish%20eqiad&h=cp1058.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1406234089&v=0&m=varnish.backend_busy&vl=N%2Fs&ti=Backend%20conn.%20too%20many&z=large [20:37:33] maybe backend_busy is the better metric? [20:37:39] yup, ^^ [20:38:14] * gwicke wished we had some default alerts on this kind of thing [20:49:37] (03CR) 10Ottomata: [C: 032 V: 032] Update wikimetrics module, run with higher concurrency in prod [operations/puppet] - 10https://gerrit.wikimedia.org/r/149115 (owner: 10Ottomata) [20:50:33] (03PS1) 10Jgreen: remove more classes from aluminium, the less prod puppet the better [operations/puppet] - 10https://gerrit.wikimedia.org/r/149116 [20:52:08] Reedy: https://gerrit.wikimedia.org/r/#/c/148593/ [20:52:13] did you deploy this, in the end? [20:52:24] (03CR) 10Jgreen: [C: 032 V: 031] remove more classes from aluminium, the less prod puppet the better [operations/puppet] - 10https://gerrit.wikimedia.org/r/149116 (owner: 10Jgreen) [20:54:31] odder: yes [20:55:54] (03CR) 10Odder: "(Reedy fixed this.)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148593 (https://bugzilla.wikimedia.org/67298) (owner: 10Brian Wolff) [20:59:02] PROBLEM - Exim SMTP on aluminium is CRITICAL: Connection timed out [20:59:03] o hi Reedy [20:59:28] Reedy: there are some collation changes to merge and scripts to run. will this happen anytime soon, or is it waiting for trusty? [20:59:45] (03PS1) 10BBlack: normalize_path should not read past the end of the url string [operations/puppet] - 10https://gerrit.wikimedia.org/r/149119 [21:00:34] ori: when you get a min, can you sanity-check https://gerrit.wikimedia.org/r/#/c/149119/ ? [21:00:46] Reedy: https://gerrit.wikimedia.org/r/#/c/140580/ https://gerrit.wikimedia.org/r/#/c/147922/ and there will be another one for all estonian wikis soon [21:01:22] greg-g: :-( [21:01:32] ? [21:01:41] You made me very unhappy last week. [21:01:57] greg-g: how was training :D [21:02:05] 15:20, 18 July 2014‎ Greg Grossmeier (Talk | contribs)‎ . . (17,494 bytes) (-92)‎ . . (→‎Next month: push back VE on tablets date) [21:02:08] JohnLewis: good [21:02:22] odder: don't blame/be mad at the messenger [21:02:28] And yet we reported July 31 in Tech News that went out on July 21 [21:02:31] https://meta.wikimedia.org/wiki/Tech/News/2014/30 [21:02:33] :-(( [21:02:51] odder: don't blame/be mad at the messenger [21:02:55] :) [21:03:02] at least I told you? [21:03:08] "told" and "you" [21:03:19] No, I wasn't aware the date was changed :-( [21:03:35] There was no e-mail, no nothing, nowhere :-( [21:03:53] and don't assume I didn't say anything internally about that [21:03:59] I guess it's my fault in missing your edit to the deployments page [21:04:17] How am I supposed to know what you write internally? [21:04:25] you aren't, which is what I said internally [21:04:30] (you're missing my point) [21:04:54] I'm saying assume good faith in my actions [21:05:06] greg-g: 'I said it internally' as an excuse is like saying 'Wait you didn't get the memo I was breaking into your house?' :p [21:05:17] again, you all are missing my point [21:05:20] * greg-g sighs [21:05:26] greg-g: I'm not assuming anything [21:05:28] 21:03 < odder> There was no e-mail, no nothing, nowhere :-( [21:05:28] 21:03 < greg-g> and don't assume I didn't say anything internally about that [21:05:32] read that again [21:05:59] greg-g: I get the point - I'm just joking :) [21:06:01] (03PS2) 10BBlack: normalize_path should not read past the end of the url string [operations/puppet] - 10https://gerrit.wikimedia.org/r/149119 [21:06:06] iow: don't berate the person who is doing your fighting for you [21:06:07] So this means someone is fixing bad communication? [21:06:35] I'm not berating anyone :-( just at the same stuff happening all over again :-( [21:06:48] no one can ever fix all bad communication, but some people can continuously work to make it better as much as they can. that's what I'm doing [21:06:57] odder: yes, but you said *I* made you sad [21:07:20] That was a lie. I only noticed your edit a few minutes ago. [21:07:24] :-) [21:07:25] technically all emotions are self-inflicted :) [21:07:35] bblack: :) [21:07:52] wwise wwords for bblack ;) [21:07:54] (o¨o) [21:07:58] ffor [21:08:09] YuviPanda: beat my correct :p [21:08:15] :) [21:08:36] odder: I accept your apology :) [21:08:58] greg-g: Also, any downtime re: Wikitech upgrade on 28th? [21:09:14] nothing I noticed, so maybe 10 minutes [21:09:15] oh unicode has emoticons [21:09:27] oh, 28th [21:09:28] 😻 [21:09:39] andrewbogott: any planned downtime associated with the wikitech upgrade on Monday? [21:09:54] Hope not! [21:09:59] Yep. In fact, google hangouts implements all their custom emoticons on top of unicode standard. [21:10:00] odder: ^ :) [21:10:05] andrewbogott: is this for OAuth? [21:10:12] yes [21:10:18] andrewbogott: w00t [21:10:18] On iOS for example, the render them in the native font (so they look different, but it allows using the native iOS emoticon picker) [21:10:27] YuviPanda: enabling the OAuth plugin or? [21:10:35] greg-g: yeah, enabling OAuth plugin [21:10:36] err [21:10:37] extension [21:10:40] Extension:OAuth [21:10:44] and on the web they map the characters to [21:11:22] (03PS3) 10Physikerwelt: WIP: Draft for Mathoid role [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 [21:12:08] (03PS4) 10Physikerwelt: WIP: Draft for Mathoid role [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 [21:14:01] (03CR) 10Physikerwelt: "Is there a way I can test this change?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 (owner: 10Physikerwelt) [21:15:24] (03CR) 10Physikerwelt: "Thank you for all your comments." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 (owner: 10Physikerwelt) [21:15:42] * odder reading SAL (yes, people do that sometimes.) [21:15:51] * greg-g does [21:15:55] A quick glance tells me Zuul is a big pain in the ass [21:16:15] Keep seeing 'restarting Zuul' messages every other day or so [21:16:26] hashar: This is why we should keep using Hangouts, this reason alone: http://i.imgur.com/FXMo2Rv.png [21:17:12] oh my [21:17:25] the url bar shows the Mac font [21:18:29] hashar: and this is why our community is awesome: http://i.imgur.com/EE5EBK0.png [21:18:44] They created a redirect for it [21:19:20] yeah same goes true for beer and cookie :-] [21:19:22] that is nice [21:20:58] hashar: Krinkle: https://en.wikipedia.org/wiki/💩 [21:21:36] (03CR) 10Hoo man: "Reedy: Could you please remove your -1? I would like to deploy this on Tuesday... a no-op with the current software version (so could have" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149031 (owner: 10Hoo man) [21:25:04] MatmaRex: Krinkle: python monkeys https://gist.github.com/hashar/3b935b41ef9126dc3501 :D [21:25:23] MatmaRex: nice one :] [21:26:11] enwiki is crazy https://en.wikipedia.org/wiki/Category:Redirects_from_symbols [21:26:40] hashar: tell us something we don't know? [21:26:53] :D [21:27:36] any wiki redirecting https://en.wikipedia.org/w/index.php?title=X%5E2%2By%5E2%2Bz%5E2%3Dr%5E2&redirect=no is insane :p [21:28:08] my question; who the fuck types that in, instead of the word 'sphere'. Hm. [21:28:12] hashar: i think they have a redirect for every unicode symbol [21:28:31] hashar: check out the unicode tables on-wiki (yeah, they have those too), they're fully linkifie [21:28:35] dd [21:29:06] JohnLewis: feel free to delete [21:29:49] Vogone: sure; give me deletion rights first :p [21:29:50] (03PS1) 10Milimetric: Make pool size configurable [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/149126 (https://bugzilla.wikimedia.org/68534) [21:30:09] (03PS1) 10Milimetric: Use new configurable pool sizes [operations/puppet] - 10https://gerrit.wikimedia.org/r/149127 (https://bugzilla.wikimedia.org/68534) [21:31:12] MatmaRex: yeah I often look at it :) [21:31:59] i better sleep now. have fun folks [21:36:12] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [21:36:59] (03PS3) 10Andrew Bogott: Revert "tools: remove libvips from exec_environ" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146348 (https://bugzilla.wikimedia.org/52717) (owner: 10Tim Landscheidt) [21:37:10] (03PS2) 10Hoo man: Add client settings needed for the other projects beta feature [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149031 [21:37:52] (03CR) 10Andrew Bogott: [C: 032] Revert "tools: remove libvips from exec_environ" [operations/puppet] - 10https://gerrit.wikimedia.org/r/146348 (https://bugzilla.wikimedia.org/52717) (owner: 10Tim Landscheidt) [21:41:25] (03PS1) 10Ori.livneh: admin: add deployment helper script to my env [operations/puppet] - 10https://gerrit.wikimedia.org/r/149131 [21:42:04] (03CR) 10Ori.livneh: [C: 032 V: 032] admin: add deployment helper script to my env [operations/puppet] - 10https://gerrit.wikimedia.org/r/149131 (owner: 10Ori.livneh) [21:46:00] >> "\ud83d\ude48" [21:46:01] Krinkle: (string) '🙈' [21:46:06] hashar: [21:46:49] >> ["\ud83d\ude48", "\ud83d\ude49", "\ud83d\ude4a"] [21:46:50] Krinkle: (object) ['🙈', '🙉', '🙊'] [21:47:00] Nice [21:48:52] :D [21:49:03] Krinkle: I am sure you would love python [21:49:13] Python doesn't love me. [21:49:14] though it doesn't run in browser :-( [21:49:29] Though I like how you can inspect an object [21:49:36] Krinkle: you can reach out Amir at wikimania, a pywikibot author. he is passionated [21:49:38] It's quite expressive [21:50:17] David Chan (divec; #mediawiki-visualeditor) uses it a lot as well for prototyping [21:51:12] (03PS1) 10Reedy: Enable oai auditing... [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149154 [21:51:17] (03CR) 10Dzahn: [C: 032] lucene: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/148992 (owner: 10Matanya) [21:51:19] Python is awesome [21:51:22] except for relative imports [21:51:23] those suck [21:51:40] (03PS2) 10Ori.livneh: Fix a couple warnings in beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148552 (owner: 10MaxSem) [21:51:45] (03CR) 10Ori.livneh: [C: 032] Fix a couple warnings in beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148552 (owner: 10MaxSem) [21:51:49] (03Merged) 10jenkins-bot: Fix a couple warnings in beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148552 (owner: 10MaxSem) [21:52:39] (03PS2) 10Reedy: Enable oai auditing... [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149154 [21:52:59] (03CR) 10Reedy: [C: 032] Enable oai auditing... [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149154 (owner: 10Reedy) [21:53:03] (03Merged) 10jenkins-bot: Enable oai auditing... [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149154 (owner: 10Reedy) [21:53:38] !log reedy Synchronized wmf-config/: (no message) (duration: 00m 14s) [21:55:06] https://code.google.com/p/chromium/issues/detail?id=397281 [22:00:51] (03CR) 10Dzahn: [C: 032] planet -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/148624 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [22:01:17] (03CR) 10Dzahn: [V: 032] planet -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/148624 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [22:05:51] (03CR) 10Dzahn: [C: 032] tendril -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/148618 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [22:06:16] (03CR) 10Dzahn: [V: 032] tendril -- update cipher suite list to support PFS [operations/puppet] - 10https://gerrit.wikimedia.org/r/148618 (https://bugzilla.wikimedia.org/53259) (owner: 10Chmarkine) [22:13:09] chasemp: btw, the diamond patches got merged. Not sure if I told you [22:13:23] saw that actually :) [22:13:45] chasemp: :D cool. No rush (hehe), but we should update the package sometime [22:15:11] (03PS1) 10Dzahn: add missing monitor_group pdf_eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/149177 [22:16:02] YuviPanda: it's on my list :) [22:16:29] (03PS1) 10Aaron Schulz: Increased basic job runners by 1 per server [operations/puppet] - 10https://gerrit.wikimedia.org/r/149179 [22:16:32] chasemp: cool [22:17:04] (03CR) 10Dzahn: [C: 032] add missing monitor_group pdf_eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/149177 (owner: 10Dzahn) [22:22:36] (03PS1) 10MaxSem: Enable PageImages gallery mode on Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149180 (https://bugzilla.wikimedia.org/66510) [22:22:49] (03CR) 10Gergő Tisza: "Sorry about this, there was a very similar change in the UploadWizard repo and I got confused." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/148593 (https://bugzilla.wikimedia.org/67298) (owner: 10Brian Wolff) [22:28:33] (03CR) 10Dzahn: "fixed Icinga reload:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149177 (owner: 10Dzahn) [22:36:39] (03PS1) 10Dzahn: site.pp - remove fresh tabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149181 [22:38:26] mwalker: did you really want 5120 M tmpfs? [22:38:29] or 512 [22:38:39] tmpfs_size => '5120M' [22:38:58] seems an odd number [22:41:09] mutante, for the moment yes -- that was the estimate with headroom for how much RAM we needed to get things to fit [22:41:21] gotcha, ok [22:41:24] we can give it more, but not until we retire tantalum because it has less RAM than the official OCG servers [22:42:06] makes sense, just thought it may have been typo, because 512 is so common [22:43:26] (03CR) 10Dzahn: [C: 032] site.pp - remove fresh tabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/149181 (owner: 10Dzahn) [22:44:32] (03CR) 10Ori.livneh: [C: 04-1] "Per IRC:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149119 (owner: 10BBlack) [22:47:34] nope; 5*1024 = 5120 [22:48:23] mwalker: true, i take back "odd" :) [22:59:33] (03CR) 10Tim Starling: [C: 031] bugzilla - raise max-age for STS to 1 year [operations/puppet] - 10https://gerrit.wikimedia.org/r/148285 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [23:00:05] RoanKattouw, mwalker, ori, MaxSem: Sir, Please deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140724T2300), the time has come. At your service [23:00:23] I'll do it [23:00:53] YuviPanda: No PDN there? ;) ---^ [23:01:00] RoanKattouw: PDN? [23:01:11] Please Do the Needful? [23:01:15] Power Delivery Network? [23:01:15] Or is that a different acronym? [23:01:24] (03CR) 10Dzahn: [C: 032] bugzilla - raise max-age for STS to 1 year [operations/puppet] - 10https://gerrit.wikimedia.org/r/148285 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [23:01:30] RoanKattouw: hehehe :D haven't heard the acronym, and sadly I didn't write jouncebot [23:01:35] Oh! [23:01:44] I could think of no one else who would make bots say "Sir," [23:01:44] RoanKattouw: if I get to rewrite reviewer-bot, I'll remember to include that there :) [23:01:57] RoanKattouw: PFA patchset, Please Do The Needful [23:02:00] What was the April 1st Gerrit message thing you and Chad wrote? [23:02:13] Oh, yes, PFA [23:02:13] That was it [23:02:15] yeah [23:02:16] PFA [23:02:20] I think sir was matanyas patch... [23:02:39] You know I was going to say something about that [23:02:40] (03CR) 10MaxSem: [C: 032] Enable PageImages gallery mode on Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149180 (https://bugzilla.wikimedia.org/66510) (owner: 10MaxSem) [23:02:42] it should be 'Sirs' [23:02:45] But sadly there are no female SWATters [23:02:47] (03Merged) 10jenkins-bot: Enable PageImages gallery mode on Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149180 (https://bugzilla.wikimedia.org/66510) (owner: 10MaxSem) [23:03:03] RoanKattouw: It also does that for all other deploys [23:03:04] RoanKattouw: Though, it does call K4|wandering sir for the FR deploys... [23:03:09] so it also call Katie Sir :P [23:03:14] haha [23:03:16] Well both Katies actually [23:03:16] https://github.com/mattofak/jouncebot/pull/4 [23:03:17] Oh she actually gets notifications? [23:03:26] I mean I did realize Katie was the only female deployer [23:03:32] But I figured she was off in FR land and wouldn't get pinged [23:03:36] RoanKattouw: Which Katie? [23:03:40] Horn [23:03:40] aude [23:03:43] :D [23:03:44] Oh right [23:03:45] aude [23:03:53] FR is on the deployment calendar [23:03:54] She doesn't have deploy rights, does she? [23:03:58] Yeah [23:04:01] For a while now [23:04:07] * Reedy wonders where RoanKattouw has been [23:04:19] It's called vacation [23:04:23] It's where you turn off your brain [23:04:24] For months? :) [23:04:33] Oh no that was school [23:04:44] It's the opposite of vacation in so many ways [23:06:11] !log maxsem Synchronized wmf-config: https://gerrit.wikimedia.org/r/149180 (duration: 00m 05s) [23:06:19] Logged the message, Master [23:06:19] RoanKattouw: but you've been kind of around here [23:07:07] or did I have my brain turned of? [23:07:09] SWAT done! [23:07:20] Yeah it must've just passed me by [23:08:00] RoanKattouw: Some light reading http://ur1.ca/edq22 :D [23:10:04] dont worry; most of that are non useful icinga notifications [23:10:23] | grep -v iicinga [23:10:54] he wouldn't notice the outages that way... which would remove much of the context, so not an option [23:11:32] (03PS3) 10BBlack: normalize_path should not read past the end of the url string [operations/puppet] - 10https://gerrit.wikimedia.org/r/149119 [23:11:45] Reedy: :D [23:16:51] (03PS1) 10Hoo man: Enable Wikibase other projects links per default for ruwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149187 [23:17:06] this is not for today, but I'll deploy it myself on Tuesday :P [23:17:17] * hoo doesn't want another -1 for no reason [23:17:41] hoo: it's not for no reason [23:17:49] it was for my laziness of knowing what I shouldn't deploy [23:17:51] Reedy: It was a no-op [23:17:59] Not sure why I added you to it [23:18:11] probably though I could save myself some work on Tuesday [23:18:15] * thought [23:18:30] When laziness fires back [23:18:37] * backfires [23:18:50] I dunno if you did [23:19:02] It appears on the open list for operations/mediawiki-config ;) [23:19:27] Let's blame gerrit for not being smart enough to hide it and forget about it :D [23:21:36] I would've just deployed it :P [23:22:00] I should have said in the commit summary that it was a no-op [23:22:10] it had duplicates, but wasn't broken, just stupid :P [23:22:31] I wrote it on the train and was pretty fed up by the lousy mobile broadband I had [23:25:24] Reeeeeeedy [23:25:58] Reedy: there are open collation patches in operations/mediawiki-config that are not being deployed :( [23:26:08] Reedy: when will they be? after trusty? [23:26:13] !log created gerrit project for jouncebot [23:26:18] Logged the message, Master [23:26:27] (or at least tell me that you don't know or something) [23:35:37] (03CR) 10BryanDavis: "You can test in labs by making a host a self-hosted puppet master and cherry picking your patch there. You can tell a host to apply a cert" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/148836 (owner: 10Physikerwelt) [23:36:14] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 15:33:13 UTC [23:37:10] AaronSchulz, ori: https://gdash.wikimedia.org/dashboards/jobq/ [23:37:18] Weird spikes. [23:38:49] (03CR) 10Ori.livneh: [C: 04-1] "you never terminate destBuffer if the URL contains no percent-encoded chars (or none of the ones you check for)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/149119 (owner: 10BBlack) [23:42:45] (03CR) 10Dzahn: ""This server supports HTTP Strict Transport Security with long duration."" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148285 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [23:53:14] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Thu 24 Jul 2014 21:52:58 UTC [23:53:54] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Thu Jul 24 23:53:51 UTC 2014