[00:09:22] !log Set email of global account "Classicfilms" to the commonswiki, enwikiquote and enwiktionary accounts of the same name. [00:09:29] Logged the message, Master [00:55:20] (03PS6) 10Jforrester: $wgContentHandlerUseDB true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170129 (https://phabricator.wikimedia.org/T51193) (owner: 10Spage) [00:56:54] springle: ping? A bunch of centralauth db queries failed with errors like Error: 1205 Lock wait timeout exceeded; try restarting transaction (10.64.16.22) [01:19:46] have you tried restarting? [01:33:53] (03PS1) 10Florianschmidtwelzow: mediawikiwiki: Add Api: and Skin: namespace to wgContentNamespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184016 (https://phabricator.wikimedia.org/T86391) [01:34:47] (03PS2) 10Florianschmidtwelzow: mediawikiwiki: Add Api: and Skin: namespace to wgContentNamespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184016 (https://phabricator.wikimedia.org/T86391) [02:17:24] !log l10nupdate Synchronized php-1.25wmf13/cache/l10n: (no message) (duration: 00m 04s) [02:17:29] !log LocalisationUpdate completed (1.25wmf13) at 2015-01-10 02:17:29+00:00 [02:17:35] Logged the message, Master [02:17:40] Logged the message, Master [02:24:45] !log l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [02:24:49] !log LocalisationUpdate completed (1.25wmf14) at 2015-01-10 02:24:48+00:00 [02:24:49] Logged the message, Master [02:24:53] Logged the message, Master [04:03:52] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 10 04:03:52 UTC 2015 (duration 3m 50s) [04:04:00] Logged the message, Master [04:38:46] (03CR) 10KartikMistry: [C: 031] "Important for https://gerrit.wikimedia.org/r/#/c/181546/" [dns] - 10https://gerrit.wikimedia.org/r/183888 (https://phabricator.wikimedia.org/T76200) (owner: 10Alexandros Kosiaris) [06:12:45] (03PS1) 10BryanDavis: logstash: Update apache2 parsing pattern [puppet] - 10https://gerrit.wikimedia.org/r/184112 [06:29:07] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:26] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:27] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:36] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:06] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:17] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:56] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:54] 3operations, Wikimedia-General-or-Unknown, WMF-Legal: Default license for operations/puppet - https://phabricator.wikimedia.org/T67270#968046 (10faidon) I realize a work's size doesn't say much about its copyrightability but just to show what's at stake here, lines of code (including whitespace and also includin... [06:36:15] (03PS12) 10KartikMistry: Content Translation configuration for Production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181546 [06:45:57] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:46:17] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:46:17] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:46:37] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:46:56] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:47:16] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:37] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:55:26] PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 685 [10:00:26] RECOVERY - check_mysql on db1008 is OK: Uptime: 7504406 Threads: 2 Questions: 208628313 Slow queries: 53241 Opens: 139887 Flush tables: 2 Open tables: 64 Queries per second avg: 27.800 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [11:04:45] (03PS1) 10Eranroz: Tel-Hai Academic College event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184119 [11:15:08] PROBLEM - Varnishkafka Delivery Errors per minute on cp3015 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [20000.0] [11:16:34] (03CR) 10TTO: [C: 04-1] Tel-Hai Academic College event (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184119 (owner: 10Eranroz) [11:19:32] (03PS2) 10Eranroz: Tel-Hai Academic College event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184119 [11:20:13] (03CR) 10Eranroz: Tel-Hai Academic College event (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184119 (owner: 10Eranroz) [11:23:46] RECOVERY - Varnishkafka Delivery Errors per minute on cp3015 is OK: OK: Less than 1.00% above the threshold [0.0] [12:15:27] PROBLEM - Varnishkafka Delivery Errors per minute on cp3003 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [20000.0] [12:21:37] RECOVERY - Varnishkafka Delivery Errors per minute on cp3003 is OK: OK: Less than 1.00% above the threshold [0.0] [13:39:53] (03PS1) 10Steinsplitter: Changing logo for klwiki from $stdlogo to commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184121 (https://phabricator.wikimedia.org/T86313) [16:03:48] (03PS1) 10Glaisher: Create 'Draft' (118) namespace on hewikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184130 (https://phabricator.wikimedia.org/T86329) [16:07:09] (03PS3) 10Aklapper: Tel-Hai Academic College event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184119 (https://phabricator.wikimedia.org/T85773) (owner: 10Eranroz) [16:32:04] (03CR) 10Yuvipanda: Add report that verifies view definitions (032 comments) [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182848 (https://phabricator.wikimedia.org/T85473) (owner: 10Yuvipanda) [16:32:56] (03PS7) 10Yuvipanda: Add report that verifies view definitions [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182848 (https://phabricator.wikimedia.org/T85473) [16:33:49] valhallasw`cloud: ^ [16:33:54] * YuviPanda addressed all your comments, I think [16:36:59] valhallasw`cloud: and seems to work fine! [16:45:27] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [16:47:16] (03PS3) 10Yuvipanda: Remove hashs, pif_edits & povwatch* view from whitelist [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/183872 (https://phabricator.wikimedia.org/T85867) [16:50:36] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [16:58:02] YuviPanda: they can't be anything that implements __ne__, as that would break your yaml output, I think [16:58:21] valhallasw`cloud: true, but it can be strings, integers, etc? [16:58:27] i mean, no reason it can’t be an int [16:58:27] mmm maybe, yeah [16:59:18] (03PS1) 10Yuvipanda: Rename some date model keys to be clearer [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184131 [16:59:27] valhallasw`cloud: so I’d say let it be duck typed and be :D also YAML does output arbitrary types [16:59:32] valhallasw`cloud: renamed things to be clearer ^ [17:00:29] YuviPanda: diff_iters is confusing [17:00:39] diff suggests it should give 'added' and 'removed' [17:00:43] hmm [17:00:47] difference? [17:01:45] YuviPanda: that's formulating it in set language, which I think is confusing as well [17:02:01] yes, it's a set difference. That doesn't have the same ring to it as 'in A but not in B' [17:02:35] valhallasw`cloud: right, it is confusing since we also have _diff [17:02:39] valhallasw`cloud: suggestions for a better name? [17:02:50] I suspect I”ll be doing that a lot, don’t want to keep writing out the entire thing [17:02:55] YuviPanda: not sure. inAbutnotB doesn't work that well :-p [17:03:02] yes :D [17:03:05] it doesn't [17:03:22] valhallasw`cloud: also I do want it to be fastish. It right now takes about 5mins or so per db... [17:03:26] well, per db host [17:03:33] YuviPanda: so? [17:03:42] so don’t want to do the list comprehension :P [17:03:45] ok, lame reason [17:03:56] YuviPanda: go sit in a corner [17:04:06] YuviPanda: have you profiled? [17:04:14] Is that the bottle neck? [17:04:26] valhallasw`cloud: I haven’t, which is why I’m in a corner [17:04:31] :> [17:04:37] anyway [17:04:45] mauybe just let it return {'added': , 'removed'} ? [17:04:52] you typically want both anyway [17:04:53] ah, hmm [17:05:03] so that’s a - b and b - a [17:05:09] yeah [17:05:14] valhallasw`cloud: I could return a tuple [17:05:21] dict. please. [17:05:26] no magic [0] [1] :P: [17:05:33] well, it will be deconstructed [17:05:41] added, removed = diff_iters(a, b) [17:06:18] (03CR) 10Merlijn van Deen: [C: 04-1] Add report that verifies view definitions (031 comment) [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182848 (https://phabricator.wikimedia.org/T85473) (owner: 10Yuvipanda) [17:06:22] YuviPanda|corner: mmm [17:06:26] YuviPanda|corner: that works, I gues [17:06:28] s [17:08:08] valhallasw`cloud: cool [17:09:06] (03CR) 10QChris: "> Lemme know if/when you want me to merge. I"d rather do this while we are both available." [puppet] - 10https://gerrit.wikimedia.org/r/172201 (https://bugzilla.wikimedia.org/72740) (owner: 10QChris) [17:10:33] (03PS2) 10Yuvipanda: Rename some date model keys to be clearer [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184131 [17:12:13] YuviPanda|corner: wait what [17:12:18] condition had two different meanings?!?! [17:12:36] (03CR) 10Yuvipanda: Add report that verifies view definitions (031 comment) [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182848 (https://phabricator.wikimedia.org/T85473) (owner: 10Yuvipanda) [17:12:55] valhallasw`cloud: I cleaned it up in https://gerrit.wikimedia.org/r/184131 [17:13:08] oh, where and condition [17:13:09] (03PS1) 10Glaisher: Remove FlaggedRevs config on metawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184133 (https://phabricator.wikimedia.org/T86443) [17:13:11] null_if and include_only_if [17:13:17] ohhhhh [17:13:28] one is a column property the other is a table property [17:13:52] YuviPanda|corner: could you move the include_only_if up to before column:? [17:14:05] then it's much clearer it's a table prop instead of a prop of the last column [17:14:15] or more than 2 spaces indentation ;D [17:14:54] valhallasw`cloud: oh yeah, that’s true :) These were autogenerated. [17:14:56] let me move it around [17:15:20] YuviPanda|corner: and maybe include_row_if instead of include_only_if? [17:15:38] valhallasw`cloud: hmm, yeah. [17:17:55] arghghghgh [17:18:00] logging is driving me crazy [17:18:07] so damn incomplete [17:20:24] (03PS4) 10Yuvipanda: Remove hashs, pif_edits & povwatch* view from whitelist [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/183872 (https://phabricator.wikimedia.org/T85867) [17:20:26] (03PS3) 10Yuvipanda: Rename some date model keys to be clearer [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184131 [17:20:28] (03PS8) 10Yuvipanda: Add report that verifies view definitions [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182848 (https://phabricator.wikimedia.org/T85473) [17:20:37] valhallasw`cloud: ^ [17:24:17] valhallasw`cloud: I’m going to profile once I merge these :) [17:25:02] YuviPanda|corner: personal image filter never became anything, I guess? [17:25:21] valhallasw`cloud: hmm? It was before my time, and I think it didn’t, no :) [17:25:34] valhallasw`cloud: legoktm or someone else from core team should go through them alll.. [17:25:37] YuviPanda|corner: labsdb-auditor schould go here, no? [17:25:53] valhallasw`cloud: I think it should go in -labs [17:25:54] both of ‘em [17:25:57] oh, right. [17:26:03] both? [17:26:08] bugs and code [17:29:42] (03CR) 10Merlijn van Deen: [C: 031] Remove hashs, pif_edits & povwatch* view from whitelist [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/183872 (https://phabricator.wikimedia.org/T85867) (owner: 10Yuvipanda) [17:30:34] (03PS4) 10Merlijn van Deen: Rename some date model keys to be clearer [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184131 (owner: 10Yuvipanda) [17:31:26] valhallasw`cloud: pfft, just +2? [17:31:34] YuviPanda|corner: I don't have +2? :P [17:31:43] SillyPanda [17:31:44] valhallasw`cloud: oh, damn [17:32:36] valhallasw`cloud: try now [17:32:44] (03CR) 10Merlijn van Deen: [C: 04-1] "Maybe we can just kill 'whitelisted' altogether? If it's in the yaml without condition, it's allowed, if there's a condition, that conditi" [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184131 (owner: 10Yuvipanda) [17:33:37] (03CR) 10Merlijn van Deen: [C: 032] Remove hashs, pif_edits & povwatch* view from whitelist [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/183872 (https://phabricator.wikimedia.org/T85867) (owner: 10Yuvipanda) [17:33:50] valhallasw`cloud: you need to v+2 and hit submit also :D [17:33:55] :< [17:34:12] (03CR) 10Merlijn van Deen: [V: 032] Remove hashs, pif_edits & povwatch* view from whitelist [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/183872 (https://phabricator.wikimedia.org/T85867) (owner: 10Yuvipanda) [17:34:19] YuviPanda|corner: no submit rights :P [17:34:19] valhallasw`cloud: hmm, if I remove whitelisted: True, it looks slightly weird [17:34:26] YuviPanda|corner: why? [17:34:33] then it's just a list of column names :-p; [17:34:34] valhallasw`cloud: I mean, the file might look weird [17:34:38] lame reason again [17:34:44] fine [17:34:51] YuviPanda|corner: wait [17:34:57] oh, wait [17:35:04] there's whitelisted.yaml AND greylisted.yaml?! [17:35:08] valhallasw`cloud: yes :P [17:35:16] valhallasw`cloud: to make things more natural to write [17:35:25] oh, for the checks you mean [17:35:29] yeah [17:35:31] but [17:35:36] I could unify them [17:35:43] and make whitelisted: True optional [17:35:56] greylisted.yaml is 'tables where some columns are greylisted'? [17:36:18] valhallasw`cloud: yup [17:36:25] I see [17:36:56] valhallasw`cloud: but I could unify the syntax. I’d still want to keep two files, tho [17:37:10] YuviPanda|corner: don't really get why, but sure [17:41:29] valhallasw`cloud: the whitelisted: true change is more involved [17:41:35] valhallasw`cloud: I think I’ll do that later? [17:41:40] ok [17:41:53] (03CR) 10Merlijn van Deen: [C: 032 V: 032] Rename some date model keys to be clearer [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184131 (owner: 10Yuvipanda) [17:41:58] YuviPanda|corner: still no submit righjts :P [17:42:01] valhallasw`cloud: :D MERGE ALL! :D \o/ [17:42:03] valhallasw`cloud: damn [17:42:28] valhallasw`cloud: there [17:42:28] now [17:42:49] (03PS1) 10Hoo man: Bug54847.php: Replace removed CentralAuthUser::getPasswordHash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184135 [17:42:51] (03PS1) 10Hoo man: Remove no longer needed hook handlers from Bug54847 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184136 [17:43:47] (03CR) 10Merlijn van Deen: [C: 032 V: 032] Add report that verifies view definitions [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/182848 (https://phabricator.wikimedia.org/T85473) (owner: 10Yuvipanda) [17:44:14] (03PS1) 10QChris: Remove unused udp2log awk scripts that forwarded to universities [puppet] - 10https://gerrit.wikimedia.org/r/184137 [17:54:32] (03PS1) 10Yuvipanda: Use absolute imports instead of relative imports [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184139 [17:54:41] (03CR) 10Reedy: [C: 031] Bug54847.php: Replace removed CentralAuthUser::getPasswordHash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184135 (owner: 10Hoo man) [17:54:52] (03PS2) 10Yuvipanda: Use absolute imports instead of relative imports [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184139 [17:56:17] PROBLEM - Varnishkafka Delivery Errors per minute on cp3009 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [20000.0] [17:57:47] valhallasw`cloud: ^ merge? :) [17:58:31] (03PS1) 10Reedy: Parameter type hints for Bug54847.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184141 [17:58:33] YuviPanda|corner: "since python's relative 9 [17:58:33] imports are fraught with errors." nah [17:58:42] YuviPanda|corner: just make real entry points! [17:58:43] ;-) [17:58:49] pfft :P [17:58:54] YuviPanda|corner: also, maybe not call it auditor but something less generic, then [17:58:54] valhallasw`cloud: well, if I wanted to run cProfile [17:59:09] I can’t do python -m cProfile -m auditor.audit [17:59:11] org.wikimedia.labs.auditor! [/java-mode] [17:59:18] that's odd [18:00:32] valhallasw`cloud: yeah, so unsure how to run cProfiler with that [18:02:02] valhallasw`cloud: labsdb.auditor, I guess [18:02:26] and then it can have labsdb.otherthings also [18:02:27] RECOVERY - Varnishkafka Delivery Errors per minute on cp3009 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:45] YuviPanda|corner: maybe, yeah [18:06:59] YuviPanda|corner: meh, relative imports indeed are not quite as well thought out as one would hope [18:07:05] yup [18:07:12] I’ve had nothing but pain *every* time I’ve tried to use them [18:07:26] I think i shall keep away from them [18:07:37] OuKB: I left comments on your hierator patch :D [18:07:43] I can fix them myself too if you would like [18:08:11] yep, i've seen [18:08:31] couldn't get to it before weekend [18:09:39] YuviPanda|corner: python -c "import cProfile; import auditor.audit; cProfile.run(...)" maybe, but that also sucks [18:09:49] valhallasw`cloud: yup [18:09:53] it does suck [18:09:58] also cProfile.run… what? [18:10:06] import auditor.audit runs the code :P [18:10:22] YuviPanda|corner: http://bugs.python.org/issue21862 [18:10:31] YuviPanda|corner: well, it shouldn't do that :P [18:10:49] !log Ran mysql:wikiadmin@db1033 [centralauth]> DELETE FROM bug_54847_password_resets WHERE r_username = 'Tk'; [18:10:49] def main() all the things! [18:10:53] Logged the message, Master [18:10:54] tch tch [18:11:02] idk, def main() always felt somewhat javaish to me [18:11:14] and I’ve never been sure what exactly to put in def main() and what not to [18:12:16] valhallasw`cloud: btw, I profiled. about 40% of time is spent in mysql calls, and 50% in parsing [18:12:24] the parsing could definitely be a lot faster [18:13:01] I bet I can make that much faster with a simple memoization :) [18:13:37] (03CR) 10Hoo man: "Please rebase upon https://gerrit.wikimedia.org/r/#/c/184136/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184141 (owner: 10Reedy) [18:13:53] hoo: :P [18:13:59] I saw that commit after I created it [18:15:40] (03CR) 10Hoo man: [C: 032] Bug54847.php: Replace removed CentralAuthUser::getPasswordHash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184135 (owner: 10Hoo man) [18:15:44] (03Merged) 10jenkins-bot: Bug54847.php: Replace removed CentralAuthUser::getPasswordHash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184135 (owner: 10Hoo man) [18:16:16] The authenticity of host '[gerrit.wikimedia.org]:29418 ([2620:0:861:3:208:80:154:81]:29418)' can't be established. [18:16:16] RSA key fingerprint is dc:e9:68:7b:99:1b:27:d0:f9:fd:ce:6a:2e:bf:92:e1. [18:17:17] PROBLEM - Varnishkafka Delivery Errors per minute on cp3008 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [20000.0] [18:18:51] !log hoo Synchronized wmf-config/Bug54847.php: Bug54847.php: Replace removed CentralAuthUser::getPasswordHash (duration: 00m 06s) [18:18:54] Logged the message, Master [18:19:17] PROBLEM - Varnishkafka Delivery Errors per minute on cp3017 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [20000.0] [18:19:36] PROBLEM - Varnishkafka Delivery Errors per minute on cp3005 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [20000.0] [18:20:47] PROBLEM - Varnishkafka Delivery Errors per minute on cp3006 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [20000.0] [18:21:46] PROBLEM - HHVM rendering on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:56] PROBLEM - Apache HTTP on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:27] RECOVERY - Varnishkafka Delivery Errors per minute on cp3008 is OK: OK: Less than 1.00% above the threshold [0.0] [18:23:36] PROBLEM - Varnishkafka Delivery Errors per minute on cp3010 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [20000.0] [18:25:31] (03PS1) 10Yuvipanda: Fix refactoring snafu [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184142 [18:25:33] (03PS1) 10Yuvipanda: Memoize parsing View definitions into Table objects [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184143 [18:25:36] RECOVERY - Varnishkafka Delivery Errors per minute on cp3005 is OK: OK: Less than 1.00% above the threshold [0.0] [18:27:57] RECOVERY - Varnishkafka Delivery Errors per minute on cp3006 is OK: OK: Less than 1.00% above the threshold [0.0] [18:28:26] PROBLEM - HHVM busy threads on mw1126 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [86.4] [18:28:46] PROBLEM - HHVM queue size on mw1126 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [80.0] [18:28:56] RECOVERY - Varnishkafka Delivery Errors per minute on cp3017 is OK: OK: Less than 1.00% above the threshold [0.0] [18:30:47] RECOVERY - Varnishkafka Delivery Errors per minute on cp3010 is OK: OK: Less than 1.00% above the threshold [0.0] [18:33:57] (03PS2) 10Yuvipanda: Memoize parsing View definitions into Table objects [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184143 [18:39:25] valhallasw`cloud: ^ gives me quite a speedup, in preliminary profiling :) [18:39:33] I’m letting it run through so I can get more complete numbers now [18:39:43] in the meantime, sleeeep [18:44:17] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 1 failures [18:48:56] PROBLEM - Varnishkafka Delivery Errors per minute on cp3010 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [20000.0] [18:58:36] RECOVERY - Varnishkafka Delivery Errors per minute on cp3010 is OK: OK: Less than 1.00% above the threshold [0.0] [19:01:16] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:06:17] PROBLEM - Varnishkafka Delivery Errors per minute on cp3017 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [20000.0] [19:07:24] valhallasw`cloud: woah, [19:07:26] without patch [19:07:26] user 3m52.699s [19:07:31] with [19:07:32] user 0m11.001s [19:08:43] well, not unexpected [19:09:00] and of course, to *really* speed it up I’ve to multithread / async it. [19:09:04] but another day. [19:09:23] (03PS3) 10Yuvipanda: Memoize parsing View definitions into Table objects [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184143 [19:13:27] RECOVERY - Varnishkafka Delivery Errors per minute on cp3017 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:57] PROBLEM - Varnishkafka Delivery Errors per minute on cp3009 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [20000.0] [19:30:36] PROBLEM - Varnishkafka Delivery Errors per minute on cp3006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [20000.0] [19:35:07] RECOVERY - Varnishkafka Delivery Errors per minute on cp3009 is OK: OK: Less than 1.00% above the threshold [0.0] [19:40:16] RECOVERY - Varnishkafka Delivery Errors per minute on cp3006 is OK: OK: Less than 1.00% above the threshold [0.0] [19:42:20] (03PS1) 10Hoo man: Don't use protected CentralAuthUser::getPasswordObject [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184147 [19:43:21] (03CR) 10Hoo man: [C: 032] Don't use protected CentralAuthUser::getPasswordObject [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184147 (owner: 10Hoo man) [19:43:25] (03Merged) 10jenkins-bot: Don't use protected CentralAuthUser::getPasswordObject [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184147 (owner: 10Hoo man) [19:44:14] !log hoo Synchronized wmf-config/Bug54847.php: Don't use protected CentralAuthUser::getPasswordObject (duration: 00m 06s) [19:44:19] Logged the message, Master [19:46:46] (03CR) 10Merlijn van Deen: [C: 04-1] "As for the copy functions: please use the https://docs.python.org/2/library/copy.html -suggested def __copy__ and def __deepcopy__ instea" (031 comment) [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184143 (owner: 10Yuvipanda) [19:51:41] (03CR) 10Yuvipanda: "Hmm, wrote explicit methods to copy because it is a cyclic data structure. However, as I write this I realize there's no need for it to b" [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/184143 (owner: 10Yuvipanda) [20:44:49] 3MediaWiki-General-or-Unknown, operations, Services, Wikidata, Analytics, wikidata-query-service: Reliable publish / subscribe event bus - https://phabricator.wikimedia.org/T84923#968636 (10JanZerebecki) http://www.fedmsg.com might fit this need. It is used/developed by Fedora and Debian people and is a federate... [21:04:08] (03PS1) 10QChris: Add logs from 'misc' caches to kafka pipeline [puppet] - 10https://gerrit.wikimedia.org/r/184183 [21:31:07] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 1 failures [21:48:07] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [22:10:32] (03CR) 10QChris: "udp2log's 5xx TSVs contain lines from the misc caches, so" [puppet] - 10https://gerrit.wikimedia.org/r/184183 (owner: 10QChris) [22:58:37] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: puppet fail [22:58:37] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: puppet fail [22:58:46] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 7 failures [22:58:46] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 12 failures [22:58:57] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 2 failures [22:58:57] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: puppet fail [22:58:57] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 36 failures [22:58:57] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 58 failures [22:58:57] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Puppet has 19 failures [22:58:57] PROBLEM - puppet last run on radon is CRITICAL: CRITICAL: puppet fail [22:58:58] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 23 failures [22:58:58] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: puppet fail [22:59:06] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 17 failures [22:59:06] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: puppet fail [22:59:06] PROBLEM - puppetmaster backend https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [22:59:07] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: puppet fail [22:59:07] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 11 failures [22:59:16] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 32 failures [22:59:17] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 12 failures [22:59:17] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: puppet fail [22:59:17] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 44 failures [22:59:18] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 32 failures [22:59:26] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: puppet fail [22:59:26] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: puppet fail [22:59:27] PROBLEM - puppet last run on elastic1025 is CRITICAL: CRITICAL: Puppet has 25 failures [22:59:27] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: puppet fail [22:59:38] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: Puppet has 9 failures [22:59:38] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 18 failures [22:59:39] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: puppet fail [22:59:46] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: puppet fail [22:59:47] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 12 failures [22:59:47] PROBLEM - puppet last run on es2006 is CRITICAL: CRITICAL: Puppet has 12 failures [22:59:47] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 50 failures [22:59:47] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: puppet fail [22:59:47] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 11 failures [22:59:47] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Puppet has 9 failures [22:59:48] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: puppet fail [22:59:56] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 71 failures [22:59:56] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 34 failures [22:59:57] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: puppet fail [22:59:57] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: puppet fail [22:59:57] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 27 failures [22:59:57] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 40 failures [22:59:57] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 16 failures [22:59:58] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: puppet fail [22:59:58] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 21 failures [23:00:02] icinga-wm: drama queen! [23:00:06] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: puppet fail [23:00:06] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 36 failures [23:00:06] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: puppet fail [23:00:06] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 31 failures [23:00:06] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet has 81 failures [23:00:07] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 61 failures [23:00:07] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: puppet fail [23:00:08] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 35 failures [23:00:16] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 12 failures [23:00:16] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 11 failures [23:00:17] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: puppet fail [23:00:17] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: puppet fail [23:00:17] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 17 failures [23:00:17] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: puppet fail [23:00:17] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 23 failures [23:00:18] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 22 failures [23:00:26] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 36 failures [23:00:26] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: puppet fail [23:00:26] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: puppet fail [23:00:26] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: puppet fail [23:00:26] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: puppet fail [23:00:27] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: puppet fail [23:00:27] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: puppet fail [23:00:36] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 18 failures [23:00:46] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: puppet fail [23:00:47] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Puppet has 50 failures [23:00:47] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: puppet fail [23:00:47] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Puppet has 13 failures [23:00:47] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: puppet fail [23:00:47] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: puppet fail [23:00:47] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: puppet fail [23:00:56] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: Puppet has 11 failures [23:00:57] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: puppet fail [23:00:57] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: puppet fail [23:00:57] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 76 failures [23:01:06] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: puppet fail [23:01:07] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: puppet fail [23:01:07] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet has 16 failures [23:01:07] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: puppet fail [23:01:07] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: puppet fail [23:01:07] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [23:01:07] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: puppet fail [23:01:08] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: puppet fail [23:01:17] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Puppet has 10 failures [23:01:17] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 74 failures [23:01:17] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: puppet fail [23:01:17] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: puppet fail [23:01:18] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 73 failures [23:01:20] (03PS1) 10Greg Grossmeier: Move notices to -releng from -qa [puppet] - 10https://gerrit.wikimedia.org/r/184199 (https://phabricator.wikimedia.org/T86053) [23:01:26] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: puppet fail [23:01:27] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: puppet fail [23:01:27] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: puppet fail [23:01:27] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 78 failures [23:01:27] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: puppet fail [23:01:27] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 71 failures [23:01:28] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: puppet fail [23:01:28] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 20 failures [23:01:28] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 38 failures [23:01:29] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: puppet fail [23:01:29] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 18 failures [23:01:36] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Puppet has 31 failures [23:01:36] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 36 failures [23:01:37] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 64 failures [23:01:46] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: puppet fail [23:01:46] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 66 failures [23:01:46] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: puppet fail [23:01:47] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 34 failures [23:01:47] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 21 failures [23:01:47] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: puppet fail [23:01:47] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: puppet fail [23:01:48] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 11 failures [23:01:56] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 26 failures [23:01:56] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 13 failures [23:01:56] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 18 failures [23:01:57] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: puppet fail [23:01:57] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: puppet fail [23:01:57] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Puppet has 17 failures [23:01:57] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: puppet fail [23:01:58] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 40 failures [23:02:07] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Puppet has 8 failures [23:02:07] PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: Puppet has 7 failures [23:02:16] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 38 failures [23:02:17] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: puppet fail [23:02:17] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 35 failures [23:02:27] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: puppet fail [23:02:27] PROBLEM - puppet last run on virt1012 is CRITICAL: CRITICAL: puppet fail [23:02:27] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 13 failures [23:02:27] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: puppet fail [23:02:27] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: Puppet has 60 failures [23:02:27] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: puppet fail [23:02:36] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 22 failures [23:02:36] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: puppet fail [23:02:37] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 32 failures [23:02:37] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: puppet fail [23:02:37] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: puppet fail [23:02:46] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: puppet fail [23:02:46] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: puppet fail [23:02:46] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Puppet has 24 failures [23:02:46] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 23 failures [23:02:47] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: puppet fail [23:02:47] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 20 failures [23:02:47] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: puppet fail [23:02:57] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: puppet fail [23:02:57] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: puppet fail [23:02:57] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 23 failures [23:02:57] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: puppet fail [23:02:58] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: puppet fail [23:02:58] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: puppet fail [23:02:58] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: puppet fail [23:03:06] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: puppet fail [23:03:07] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Puppet has 71 failures [23:03:16] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 15 failures [23:03:16] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 21 failures [23:03:17] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: Puppet has 17 failures [23:03:17] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: puppet fail [23:03:26] PROBLEM - puppet last run on es2003 is CRITICAL: CRITICAL: puppet fail [23:03:26] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 38 failures [23:03:26] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: puppet fail [23:03:27] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 24 failures [23:03:27] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Puppet has 41 failures [23:03:27] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: puppet fail [23:03:36] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Puppet has 17 failures [23:03:36] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: Puppet has 30 failures [23:03:36] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: puppet fail [23:03:36] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: puppet fail [23:03:37] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 69 failures [23:03:37] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: Puppet has 80 failures [23:03:37] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Puppet has 10 failures [23:03:38] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: puppet fail [23:03:46] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Puppet has 8 failures [23:03:46] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: puppet fail [23:03:47] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: puppet fail [23:03:47] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: puppet fail [23:03:47] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: puppet fail [23:03:47] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: puppet fail [23:03:57] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 12 failures [23:03:58] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: puppet fail [23:03:58] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: puppet fail [23:03:58] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 17 failures [23:03:58] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 24 failures [23:03:58] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 38 failures [23:03:59] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: puppet fail [23:03:59] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: puppet fail [23:04:08] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 68 failures [23:04:08] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 17 failures [23:04:09] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 35 failures [23:04:09] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 35 failures [23:04:09] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: puppet fail [23:04:09] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Puppet has 15 failures [23:04:09] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: puppet fail [23:04:09] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Puppet has 21 failures [23:04:10] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: puppet fail [23:04:10] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: puppet fail [23:04:11] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Puppet has 73 failures [23:04:16] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 11 failures [23:04:16] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 22 failures [23:04:17] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: puppet fail [23:04:27] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Puppet has 39 failures [23:04:27] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 40 failures [23:04:27] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: puppet fail [23:04:27] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet has 8 failures [23:04:28] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Puppet has 15 failures [23:04:36] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: puppet fail [23:04:36] PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: Puppet has 15 failures [23:04:36] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: puppet fail [23:04:37] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 11 failures [23:04:37] PROBLEM - puppet last run on es2005 is CRITICAL: CRITICAL: puppet fail [23:04:46] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: puppet fail [23:04:46] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: puppet fail [23:04:46] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 28 failures [23:04:46] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 9 failures [23:04:46] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Puppet has 28 failures [23:04:47] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: puppet fail [23:04:47] PROBLEM - HHVM busy threads on mw1126 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [86.4] [23:04:48] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 10 failures [23:04:48] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: puppet fail [23:04:49] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: puppet fail [23:04:49] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 29 failures [23:04:56] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: puppet fail [23:04:56] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: puppet fail [23:04:57] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 14 failures [23:04:57] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: puppet fail [23:04:57] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 37 failures [23:04:57] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: puppet fail [23:05:06] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 58 failures [23:05:07] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: puppet fail [23:05:07] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 25 failures [23:05:07] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 34 failures [23:05:07] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 23 failures [23:05:07] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: puppet fail [23:05:07] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 25 failures [23:05:18] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 75 failures [23:05:18] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 20 failures [23:05:18] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 32 failures [23:05:18] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 20 failures [23:05:19] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: puppet fail [23:05:19] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Puppet has 9 failures [23:05:20] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 16 failures [23:05:20] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 23 failures [23:05:20] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: puppet fail [23:05:26] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 64 failures [23:05:26] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 12 failures [23:05:26] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 13 failures [23:05:36] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: puppet fail [23:05:37] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 20 failures [23:05:37] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: puppet fail [23:05:46] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: Puppet has 24 failures [23:05:46] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: puppet fail [23:05:47] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: Puppet has 11 failures [23:05:47] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet has 61 failures [23:05:47] PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: Puppet has 13 failures [23:05:47] PROBLEM - puppet last run on labstore2001 is CRITICAL: CRITICAL: Puppet has 9 failures [23:05:47] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: puppet fail [23:05:48] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: puppet fail [23:05:48] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: puppet fail [23:05:49] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 16 failures [23:05:49] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: puppet fail [23:05:50] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 22 failures [23:05:50] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Puppet has 21 failures [23:05:51] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: Puppet has 18 failures [23:05:56] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: puppet fail [23:05:57] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: puppet fail [23:05:57] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: puppet fail [23:05:57] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 43 failures [23:06:06] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 11 failures [23:06:06] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 12 failures [23:06:07] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Puppet has 22 failures [23:06:07] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: puppet fail [23:06:07] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: puppet fail [23:06:07] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 17 failures [23:06:07] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet has 12 failures [23:06:08] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: puppet fail [23:06:16] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 24 failures [23:06:16] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 24 failures [23:06:17] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: puppet fail [23:06:17] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: puppet fail [23:06:17] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: puppet fail [23:06:17] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: puppet fail [23:06:17] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 11 failures [23:06:18] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: puppet fail [23:06:18] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: puppet fail [23:06:19] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 36 failures [23:06:19] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: puppet fail [23:06:20] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 25 failures [23:06:20] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 20 failures [23:06:26] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: puppet fail [23:06:26] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: puppet fail [23:06:26] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: puppet fail [23:06:26] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: puppet fail [23:06:27] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Puppet has 29 failures [23:06:27] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet has 82 failures [23:06:27] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: puppet fail [23:06:36] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 10 failures [23:06:36] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 21 failures [23:06:36] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Puppet has 24 failures [23:06:37] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: Puppet has 25 failures [23:06:37] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 76 failures [23:06:37] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: puppet fail [23:06:46] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Puppet has 20 failures [23:06:46] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 33 failures [23:06:46] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 26 failures [23:06:46] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 23 failures [23:06:47] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: Puppet has 9 failures [23:06:47] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: puppet fail [23:06:47] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: puppet fail [23:06:48] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: puppet fail [23:06:56] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: Puppet has 89 failures [23:06:58] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: puppet fail [23:06:59] PROBLEM - puppet last run on dbstore2001 is CRITICAL: CRITICAL: Puppet has 10 failures [23:06:59] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: puppet fail [23:06:59] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: Puppet has 7 failures [23:06:59] PROBLEM - puppet last run on capella is CRITICAL: CRITICAL: Puppet has 23 failures [23:06:59] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: Puppet has 8 failures [23:06:59] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: Puppet has 19 failures [23:06:59] PROBLEM - puppet last run on es2002 is CRITICAL: CRITICAL: Puppet has 9 failures [23:06:59] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: puppet fail [23:06:59] PROBLEM - puppet last run on es2009 is CRITICAL: CRITICAL: puppet fail [23:07:01] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: Puppet has 16 failures [23:07:01] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: puppet fail [23:07:01] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: puppet fail [23:07:01] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: puppet fail [23:07:06] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: puppet fail [23:07:06] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: puppet fail [23:07:06] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Puppet has 8 failures [23:07:06] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 26 failures [23:07:07] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 14 failures [23:07:07] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: Puppet has 26 failures [23:07:07] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: puppet fail [23:07:17] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 26 failures [23:07:17] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: puppet fail [23:07:17] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: puppet fail [23:07:18] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: puppet fail [23:07:18] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 14 failures [23:07:18] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 22 failures [23:07:18] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 29 failures [23:07:18] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: puppet fail [23:07:18] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 20 failures [23:07:26] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet has 71 failures [23:07:26] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 39 failures [23:07:26] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: puppet fail [23:07:27] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: puppet fail [23:07:27] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 10 failures [23:07:27] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 71 failures [23:07:27] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: puppet fail [23:07:28] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: puppet fail [23:07:36] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 18 failures [23:07:36] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Puppet has 17 failures [23:07:36] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 27 failures [23:07:36] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 19 failures [23:07:36] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: puppet fail [23:07:37] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 29 failures [23:07:37] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 75 failures [23:07:38] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: puppet fail [23:07:38] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: Puppet has 76 failures [23:07:39] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: puppet fail [23:07:46] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 18 failures [23:07:47] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: puppet fail [23:07:47] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: puppet fail [23:07:47] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: puppet fail [23:07:47] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: puppet fail [23:07:47] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 68 failures [23:07:56] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 36 failures [23:07:56] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 31 failures [23:07:57] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: puppet fail [23:08:07] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 15 failures [23:08:07] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 21 failures [23:08:07] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 15 failures [23:08:07] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: puppet fail [23:08:07] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: puppet fail [23:08:16] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 30 failures [23:08:17] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 59 failures [23:08:17] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: puppet fail [23:08:17] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: puppet fail [23:08:18] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 16 failures [23:08:27] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 24 failures [23:08:27] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 21 failures [23:08:27] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 12 failures [23:08:27] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Puppet has 36 failures [23:08:27] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: puppet fail [23:08:36] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: puppet fail [23:08:36] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 26 failures [23:08:36] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: puppet fail [23:08:36] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: puppet fail [23:08:36] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 23 failures [23:08:37] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 36 failures [23:08:37] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: puppet fail [23:08:38] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: puppet fail [23:08:38] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: puppet fail [23:08:39] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 67 failures [23:08:39] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: puppet fail [23:08:46] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 67 failures [23:08:47] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: puppet fail [23:08:47] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 21 failures [23:08:47] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: puppet fail [23:08:47] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 20 failures [23:08:47] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: puppet fail [23:08:57] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 12 failures [23:08:57] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 7 failures [23:08:58] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 20 failures [23:08:58] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: puppet fail [23:09:06] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 40 failures [23:09:06] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: puppet fail [23:09:06] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 21 failures [23:09:06] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 19 failures [23:09:16] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: puppet fail [23:09:17] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: puppet fail [23:09:17] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Puppet has 20 failures [23:09:17] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 14 failures [23:09:17] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: puppet fail [23:09:17] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: puppet fail [23:09:17] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Puppet has 29 failures [23:09:18] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: puppet fail [23:09:18] PROBLEM - puppet last run on db2040 is CRITICAL: CRITICAL: Puppet has 12 failures [23:09:19] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: puppet fail [23:09:26] PROBLEM - puppet last run on db2042 is CRITICAL: CRITICAL: puppet fail [23:09:26] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 41 failures [23:09:26] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: puppet fail [23:09:26] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 51 failures [23:09:26] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: puppet fail [23:09:27] PROBLEM - puppet last run on nembus is CRITICAL: CRITICAL: puppet fail [23:09:27] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 35 failures [23:09:36] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: puppet fail [23:09:36] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: puppet fail [23:09:36] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: puppet fail [23:09:36] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 12 failures [23:09:37] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: puppet fail [23:09:37] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: puppet fail [23:09:37] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: puppet fail [23:09:37] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 22 failures [23:09:38] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Puppet has 18 failures [23:09:38] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: puppet fail [23:09:46] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: puppet fail [23:09:47] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: Puppet has 15 failures [23:09:47] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: puppet fail [23:09:56] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: puppet fail [23:09:57] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 18 failures [23:09:57] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 12 failures [23:09:57] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 24 failures [23:09:57] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 16 failures [23:09:57] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [23:09:57] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: puppet fail [23:09:57] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: Puppet has 25 failures [23:09:58] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 30 failures [23:10:08] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 22 failures [23:10:08] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 71 failures [23:10:09] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: puppet fail [23:10:10] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: puppet fail [23:10:10] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 23 failures [23:10:10] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Puppet has 23 failures [23:10:10] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: puppet fail [23:10:11] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 73 failures [23:10:11] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: puppet fail [23:10:11] (03CR) 10Greg Grossmeier: "Note: I'm not entirely sure about the changes to eg" [puppet] - 10https://gerrit.wikimedia.org/r/184199 (https://phabricator.wikimedia.org/T86053) (owner: 10Greg Grossmeier) [23:10:11] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 66 failures [23:10:16] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 23 failures [23:10:16] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 24 failures [23:10:16] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Puppet has 17 failures [23:10:17] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: puppet fail [23:10:17] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: puppet fail [23:10:17] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [23:10:17] PROBLEM - puppet last run on plutonium is CRITICAL: CRITICAL: puppet fail [23:10:18] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 23 failures [23:10:26] PROBLEM - puppet last run on lithium is CRITICAL: CRITICAL: Puppet has 7 failures [23:10:27] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: puppet fail [23:10:27] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: Puppet has 24 failures [23:10:28] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: puppet fail [23:10:36] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 13 failures [23:10:36] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 74 failures [23:10:37] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet has 17 failures [23:10:37] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: puppet fail [23:10:37] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Puppet has 21 failures [23:10:37] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: puppet fail [23:10:37] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 43 failures [23:10:38] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: puppet fail [23:10:38] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: puppet fail [23:10:39] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: puppet fail [23:10:39] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: puppet fail [23:10:40] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 13 failures [23:10:40] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: puppet fail [23:10:46] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet has 13 failures [23:10:46] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 13 failures [23:10:46] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: puppet fail [23:10:47] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: puppet fail [23:10:56] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: puppet fail [23:10:56] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 8 failures [23:10:56] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 39 failures [23:10:56] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: puppet fail [23:10:56] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 34 failures [23:10:56] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: puppet fail [23:10:57] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Puppet has 70 failures [23:10:57] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Puppet has 10 failures [23:10:58] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: puppet fail [23:10:58] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: puppet fail [23:11:06] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: puppet fail [23:11:06] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Puppet has 38 failures [23:11:06] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 30 failures [23:11:06] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Puppet has 15 failures [23:11:06] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 67 failures [23:11:07] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 21 failures [23:11:07] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 32 failures [23:11:08] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Puppet has 37 failures [23:11:08] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Puppet has 18 failures [23:11:09] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Puppet has 17 failures [23:11:09] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 17 failures [23:11:10] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 18 failures [23:11:10] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 14 failures [23:11:11] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 17 failures [23:11:11] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 18 failures [23:11:12] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 22 failures [23:11:12] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: puppet fail [23:11:13] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: puppet fail [23:11:16] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: puppet fail [23:11:16] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 66 failures [23:11:16] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 11 failures [23:11:16] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Puppet has 21 failures [23:11:17] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: puppet fail [23:11:17] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: puppet fail [23:11:17] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: Puppet has 24 failures [23:11:26] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: puppet fail [23:11:26] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Puppet has 33 failures [23:11:26] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: puppet fail [23:11:26] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: puppet fail [23:11:26] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet has 23 failures [23:11:27] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: puppet fail [23:11:27] PROBLEM - puppet last run on analytics1002 is CRITICAL: CRITICAL: Puppet has 11 failures [23:11:36] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet has 66 failures [23:11:37] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet has 9 failures [23:11:37] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: Puppet has 12 failures [23:11:37] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: puppet fail [23:11:37] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: puppet fail [23:11:46] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: puppet fail [23:11:47] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: puppet fail [23:11:47] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: puppet fail [23:11:47] PROBLEM - puppet last run on db2001 is CRITICAL: CRITICAL: Puppet has 9 failures [23:11:47] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: puppet fail [23:11:47] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: Puppet has 37 failures [23:11:47] PROBLEM - puppet last run on es2004 is CRITICAL: CRITICAL: Puppet has 24 failures [23:11:48] PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: puppet fail [23:11:56] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: puppet fail [23:11:56] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: puppet fail [23:11:56] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: puppet fail [23:11:56] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Puppet has 21 failures [23:11:56] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: puppet fail [23:11:57] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Puppet has 14 failures [23:11:57] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 28 failures [23:11:58] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: puppet fail [23:11:58] PROBLEM - puppet last run on virt1010 is CRITICAL: CRITICAL: puppet fail [23:11:59] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Puppet has 27 failures [23:11:59] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: puppet fail [23:12:06] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 20 failures [23:12:06] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Puppet has 6 failures [23:12:06] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Puppet has 34 failures [23:12:07] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: puppet fail [23:12:07] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 75 failures [23:12:16] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: puppet fail [23:12:17] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet has 54 failures [23:12:17] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Puppet has 27 failures [23:12:17] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Puppet has 32 failures [23:12:17] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: puppet fail [23:12:17] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: puppet fail [23:12:17] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: puppet fail [23:12:18] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 14 failures [23:12:18] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: puppet fail [23:12:19] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: puppet fail [23:12:19] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 18 failures [23:12:20] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.117 second response time [23:12:26] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 20 failures [23:12:26] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: puppet fail [23:12:26] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: puppet fail [23:12:26] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 31 failures [23:12:27] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: puppet fail [23:12:27] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: puppet fail [23:12:27] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: puppet fail [23:12:27] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet has 35 failures [23:12:28] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: puppet fail [23:12:36] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: puppet fail [23:12:36] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: puppet fail [23:12:37] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: puppet fail [23:12:37] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Puppet has 28 failures [23:12:37] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: puppet fail [23:12:37] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: puppet fail [23:12:46] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.032 second response time [23:12:46] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 17 failures [23:12:46] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: puppet fail [23:12:46] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Puppet has 26 failures [23:12:46] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: puppet fail [23:12:47] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Puppet has 28 failures [23:12:47] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Puppet has 74 failures [23:12:57] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: Puppet has 21 failures [23:12:57] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Puppet has 21 failures [23:12:57] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 79 failures [23:12:57] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: puppet fail [23:12:58] PROBLEM - puppet last run on es2007 is CRITICAL: CRITICAL: puppet fail [23:12:58] PROBLEM - puppet last run on db2004 is CRITICAL: CRITICAL: Puppet has 19 failures [23:13:06] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: puppet fail [23:13:06] PROBLEM - puppet last run on haedus is CRITICAL: CRITICAL: Puppet has 22 failures [23:13:06] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Puppet has 20 failures [23:13:07] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: puppet fail [23:13:07] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: puppet fail [23:13:07] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: puppet fail [23:13:17] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 71 failures [23:13:17] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 74 failures [23:13:17] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: puppet fail [23:13:17] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: puppet fail [23:13:17] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 71 failures [23:13:17] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: puppet fail [23:13:18] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: puppet fail [23:13:18] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: Puppet has 34 failures [23:13:18] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 17 failures [23:13:26] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: puppet fail [23:13:26] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 13 failures [23:13:26] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: puppet fail [23:13:26] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 70 failures [23:13:27] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: puppet fail [23:13:27] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Puppet has 64 failures [23:13:27] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Puppet has 72 failures [23:13:36] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: puppet fail [23:13:36] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 30 failures [23:13:36] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: puppet fail [23:13:36] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: puppet fail [23:13:36] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: puppet fail [23:13:37] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Puppet has 26 failures [23:13:37] PROBLEM - puppet last run on analytics1001 is CRITICAL: CRITICAL: puppet fail [23:13:37] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Puppet has 14 failures [23:13:38] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: puppet fail [23:13:48] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 16 failures [23:13:48] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: puppet fail [23:13:56] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 23 failures [23:13:56] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: puppet fail [23:13:56] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: puppet fail [23:13:57] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: puppet fail [23:13:57] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: puppet fail [23:13:57] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Puppet has 33 failures [23:14:06] PROBLEM - puppet last run on lvs2003 is CRITICAL: CRITICAL: puppet fail [23:14:07] PROBLEM - puppet last run on db2003 is CRITICAL: CRITICAL: puppet fail [23:14:16] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 38 failures [23:14:16] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: puppet fail [23:14:16] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: puppet fail [23:14:17] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Puppet has 23 failures [23:14:27] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: puppet fail [23:14:28] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: puppet fail [23:14:28] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Puppet has 11 failures [23:14:28] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: puppet fail [23:14:36] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: Puppet has 5 failures [23:14:36] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Puppet has 36 failures [23:14:37] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Puppet has 13 failures [23:14:37] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: puppet fail [23:14:37] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: puppet fail [23:14:37] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: Puppet has 10 failures [23:14:46] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: puppet fail [23:14:47] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 18 failures [23:14:47] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: puppet fail [23:14:58] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: puppet fail [23:14:58] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: puppet fail [23:14:58] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: puppet fail [23:15:06] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Puppet has 10 failures [23:15:07] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: puppet fail [23:15:07] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: puppet fail [23:15:07] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: puppet fail [23:15:17] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: puppet fail [23:15:26] PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: Puppet has 7 failures [23:15:46] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:15:56] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:16:08] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:16:09] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:16:09] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:16:17] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:16:36] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:16:47] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:16:47] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:16:47] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:16:47] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:16:47] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:16:57] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:16:57] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:16:57] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:17:06] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:17:06] RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:17:06] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:17:07] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:17:07] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:17:07] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:17:18] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:17:19] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:17:20] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:17:20] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:17:20] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:17:21] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:17:26] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:17:27] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:17:36] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:17:37] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:17:37] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:17:37] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:17:37] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:17:37] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:17:46] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:17:46] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:17:47] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:17:47] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:17:47] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:17:47] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:17:47] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:17:48] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:17:48] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:17:49] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:17:57] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:17:57] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:17:57] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:17:58] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:17:58] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:18:06] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:18:06] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:18:06] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:18:06] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:18:06] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:18:06] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:18:07] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:18:07] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:18:08] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:18:16] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:18:17] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:18:17] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:18:17] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:18:17] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:18:17] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:18:18] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:18:18] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:18:18] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:18:19] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:18:19] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:18:20] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:18:26] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:18:26] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:18:26] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:18:37] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:18:38] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:18:46] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:18:46] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:18:46] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:18:46] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:18:47] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:18:47] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:18:47] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:18:48] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:18:57] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:18:57] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:18:57] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:19:06] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:19:06] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:19:07] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:19:07] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:19:07] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:19:07] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:19:16] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:19:16] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:19:17] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:19:17] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:19:17] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:19:26] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:19:27] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:19:37] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:19:37] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:19:37] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:19:37] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:19:37] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:19:38] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:19:38] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:19:46] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:19:46] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:19:46] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:19:48] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:19:48] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:19:56] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:19:56] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:19:56] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:19:57] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:20:06] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:20:07] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:20:07] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:20:07] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:20:07] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:20:07] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:20:17] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:20:17] RECOVERY - puppet last run on db2041 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:20:17] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:20:17] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:20:17] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:20:17] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:20:18] <_joe_> !log restarted the puppetmaster on palladium [23:20:24] Logged the message, Master [23:20:26] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:20:27] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:20:27] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:20:27] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:20:27] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:20:27] RECOVERY - puppet last run on virt1012 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:20:27] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:20:28] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:20:38] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:20:39] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:20:39] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:20:39] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:20:47] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:20:47] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:20:47] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:20:47] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:20:48] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:20:48] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:20:48] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:20:48] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:20:56] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:20:56] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:20:56] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:20:56] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:20:57] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:20:57] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:21:06] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:21:06] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:21:06] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:21:06] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:21:06] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:21:06] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:21:06] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:21:07] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:21:07] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:21:16] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:21:16] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:21:17] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:21:17] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:21:17] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:21:18] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:21:27] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:21:28] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:21:36] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:21:36] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:21:36] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:21:36] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:21:37] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:21:37] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:21:37] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:21:38] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:21:38] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:21:47] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:21:47] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:21:47] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:21:48] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:21:48] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:21:56] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:21:56] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:21:56] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:21:56] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:21:57] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:21:57] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:21:57] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:22:06] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:22:06] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:22:06] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:22:07] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:22:07] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:22:07] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:22:07] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:22:16] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:22:16] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:22:16] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:22:17] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:22:27] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:22:27] RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:22:27] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:22:37] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:22:37] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:22:37] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:22:37] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:22:37] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:22:37] RECOVERY - puppet last run on labstore2001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:22:37] RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:22:38] RECOVERY - puppet last run on es2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:22:46] RECOVERY - puppet last run on es2005 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:22:46] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:22:46] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:22:46] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:22:47] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:22:57] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:23:06] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:23:06] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:23:07] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:23:16] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:23:16] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:16] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:17] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:23:17] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:23:17] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:17] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:23:27] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:23:27] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:23:27] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:23:28] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:28] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:23:28] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:23:28] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:23:28] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:23:29] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:36] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:23:38] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:23:47] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:23:47] RECOVERY - puppet last run on dbstore2001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:23:47] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:23:47] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:23:57] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:23:57] RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:23:57] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:23:57] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:23:57] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:23:57] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:23:58] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:23:58] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:23:58] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:23:59] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:24:07] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:24:07] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:24:16] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:16] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:24:18] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:18] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:24:18] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:18] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:24:18] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:24:18] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:24:18] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:24:19] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:24:19] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:24:26] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:24:26] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:26] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:26] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:26] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:24:27] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:24:27] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:28] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:24:28] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:24:29] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:29] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:24:30] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:24:38] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:24:38] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:24:39] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:24:46] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:46] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:24:47] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:24:47] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:24:47] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:24:47] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:24:47] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:56] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:24:56] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:57] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:24:57] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:24:57] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:24:57] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:24:57] RECOVERY - puppet last run on capella is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:25:06] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:25:07] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:25:07] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:25:07] RECOVERY - puppet last run on es2002 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:25:07] RECOVERY - puppet last run on es2009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:07] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:25:07] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:25:08] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:25:08] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:09] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:25:09] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:25:10] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:25:10] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:25:16] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:25:17] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:25:17] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:25:17] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:25:17] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:25:26] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:26] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:25:26] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:25:27] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:25:27] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:25:27] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:25:27] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:36] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:36] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:25:36] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:25:36] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:25:37] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:25:37] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:25:37] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:25:38] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:25:47] RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:25:48] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:25:48] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:25:49] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:25:49] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:25:57] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:25:57] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:25:57] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:26:06] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:06] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:07] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:26:07] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:07] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:26:16] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:26:17] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:26:17] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:26:17] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:26:17] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:26:17] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:26:17] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:26:18] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:26:27] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:26:27] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:26:27] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:26:27] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:26:27] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:26:27] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:26:27] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:26:28] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:26:36] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:26:36] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:26:36] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:26:36] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:26:36] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:26:37] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:37] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:26:38] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:26:38] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:39] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:26:39] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:26:46] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:26:46] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:26:46] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:47] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:26:47] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:26:47] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:47] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:26:47] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:26:48] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:26:48] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:26:49] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:26:49] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:26:50] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:26:50] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:26:57] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:26:58] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:59] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:26:59] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:26:59] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:26:59] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:59] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:26:59] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:26:59] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:27:00] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:06] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:27:06] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:27:06] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:06] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:27:07] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:27:07] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:16] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:27:16] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:27:17] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:27:17] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:27:26] RECOVERY - puppet last run on db2040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:26] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:26] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:26] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:26] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:27] RECOVERY - puppet last run on db2042 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:27:27] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:27:28] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:27:28] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:27:29] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:27:36] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:27:37] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:27:37] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:27:37] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:37] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:27:37] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:37] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:46] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:47] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:47] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:27:47] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:27:47] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:27:56] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:27:56] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:27:56] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:27:56] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:27:57] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:27:57] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:27:57] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:27:58] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:27:58] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:27:59] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:27:59] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:28:00] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:28:06] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:28:06] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:28:07] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:28:07] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:28:07] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:28:16] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:28:16] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:28:16] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:16] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:28:16] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:28:17] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:28:17] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:28:18] RECOVERY - puppet last run on analytics1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:28:27] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:28:27] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:28:27] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:28:27] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:27] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:28:27] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:28:28] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:37] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:28:37] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:28:37] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:28:37] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:28:37] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:37] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:28:37] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:28:38] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:28:38] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:39] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:28:39] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:28:40] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:28:40] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:41] RECOVERY - puppet last run on es2004 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:28:41] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:28:42] RECOVERY - puppet last run on nembus is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:28:42] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:28:46] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:28:46] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:46] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:28:47] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:47] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:28:47] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:28:47] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:28:48] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:28:56] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:28:56] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:28:56] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:28:57] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:28:57] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:29:06] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:29:06] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:29:06] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:29:07] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:29:07] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:29:07] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:29:07] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:29:08] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:29:08] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:29:09] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:29:09] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:29:10] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:29:10] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:11] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:29:11] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:29:12] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:29:16] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:29:16] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:29:16] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:16] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:29:16] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:29:17] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:29:17] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:29:18] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:29:22] ... [23:29:26] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:26] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:29:26] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:29:27] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:29:27] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:29:27] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:29:27] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:29:28] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:29:36] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:29:37] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:29:37] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:29:37] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:29:37] RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:29:46] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:46] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:29:47] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:29:47] RECOVERY - puppet last run on db2004 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:29:47] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:29:47] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:29:47] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:29:48] RECOVERY - puppet last run on db2001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:29:48] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:29:56] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:56] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:29:56] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:29:56] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:29:56] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:29:57] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:57] RECOVERY - puppet last run on virt1010 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:29:58] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:29:58] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:29:59] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:29:59] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:30:06] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:07] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:07] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:30:07] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:16] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:30:16] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:30:17] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:30:17] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:30:17] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:30:18] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:30:18] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:30:18] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:26] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:30:26] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:30:26] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:26] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:26] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:30:27] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:30:27] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:30:28] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:30:28] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:30:29] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:30:29] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:30:30] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:30:36] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:30:37] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:30:38] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:38] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:30:38] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:30:38] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:30:38] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:30:46] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:30:46] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:30:46] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:30:46] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:30:46] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:30:47] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:30:47] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:30:56] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:30:57] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:30:57] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:31:06] RECOVERY - puppet last run on es2007 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:31:06] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:31:06] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:31:07] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:31:07] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:31:07] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:31:07] RECOVERY - puppet last run on haedus is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:31:08] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:31:08] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:31:09] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:31:17] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:31:17] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:31:18] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:31:18] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:31:18] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:31:18] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:31:18] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:31:18] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:31:18] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:31:26] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:31:26] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:31:26] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:31:26] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:31:27] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:31:27] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:31:27] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:31:36] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:31:36] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:31:36] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:31:37] RECOVERY - puppet last run on analytics1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:31:37] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:31:37] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:31:37] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:31:48] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:31:48] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:31:48] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:31:56] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:31:56] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:31:56] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:31:57] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:31:57] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:32:06] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:32:07] RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:32:07] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:32:16] RECOVERY - puppet last run on db2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:16] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:32:26] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:32:26] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:27] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:27] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:32:36] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:36] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:32:37] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:37] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:32:47] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:32:47] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:32:47] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:47] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:32:58] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:32:58] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:33:06] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:33:06] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:33:07] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:33:27] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:33:27] RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:33:48] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:33:57] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:34:08] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures