[00:00:39] (03PS1) 10BryanDavis: iegreview: fix hostname for logging server [puppet] - 10https://gerrit.wikimedia.org/r/167350 [00:01:35] mutante: ^ should fix the logging crash [00:01:55] (03PS1) 10Dzahn: iegreview - include app passwords from private [puppet] - 10https://gerrit.wikimedia.org/r/167351 [00:02:22] doh. good find [00:03:09] (03CR) 10Dzahn: [C: 032] iegreview: fix hostname for logging server [puppet] - 10https://gerrit.wikimedia.org/r/167350 (owner: 10BryanDavis) [00:03:45] (03CR) 10Dzahn: [C: 032] iegreview - include app passwords from private [puppet] - 10https://gerrit.wikimedia.org/r/167351 (owner: 10Dzahn) [00:05:27] bd808: worked:) i saw it add the passwords to config [00:05:40] how about the logging [00:05:41] excellent [00:06:05] The requested URL /index was not found on this server. [00:06:12] missing file ext? [00:06:31] yea, should be index.php [00:06:45] no that's the router script [00:06:55] I'll see if I got a log message [00:07:13] https://iegreview.wikimedia.org/ redirects to /index [00:07:29] compare https://iegreview.wikimedia.org/index.php/index [00:08:24] so maybe my rewrite rule is broken? [00:08:28] Call to undefined function Wikimedia\\IEGReview\\curl_init() in /srv/deployment/iegreview/iegreview/src/ParsoidClient.php [00:08:52] argh [00:08:55] ok that's an app bug [00:09:06] I wonder why it works at http://iegreview.wmflabs.org/index [00:09:34] must be php version differences between precise and trusty [00:09:58] Easy enough to fix I think. But we are good for today [00:10:28] We have proven that most things work and if we get past this we will run into the db not being ready [00:10:51] that is true, yes [00:11:12] and the rewrite rule shouldn't be a big issue either [00:36:25] (03CR) 10Dzahn: "MAC address ok, talks to carbon, but DHCPDISCOVER from 84:2b:2b:fd:bd:0e via 10.64.0.2: network 10.64.0.0/22: no free leases" [puppet] - 10https://gerrit.wikimedia.org/r/167149 (owner: 10Dzahn) [00:51:35] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 301 seconds [00:52:15] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 349 seconds [00:53:25] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 12 seconds [00:53:48] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -1 seconds [00:59:33] (03PS3) 10Gage: Enable GELF for MRAppManager part 2 [puppet] - 10https://gerrit.wikimedia.org/r/167044 [02:20:20] !log LocalisationUpdate completed (1.25wmf3) at 2014-10-18 02:20:20+00:00 [02:20:32] Logged the message, Master [02:33:01] !log LocalisationUpdate completed (1.25wmf4) at 2014-10-18 02:33:01+00:00 [02:33:06] Logged the message, Master [03:51:18] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 18 03:51:18 UTC 2014 (duration 51m 17s) [03:51:23] Logged the message, Master [06:27:59] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:21] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: puppet fail [06:28:30] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: puppet fail [06:28:31] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: puppet fail [06:28:39] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: puppet fail [06:28:40] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: puppet fail [06:28:50] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.087 second response time [06:29:09] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: puppet fail [06:29:20] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [06:30:49] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:00] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:00] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:09] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 3 failures [06:31:09] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:10] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:10] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:19] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:20] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:29] PROBLEM - OCG health on ocg1002 is CRITICAL: CRITICAL: ocg_job_status 311948 msg: ocg_render_job_queue 805 msg (=500 critical) [06:31:30] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:30] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:30] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:31] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:39] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:39] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:39] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:40] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:40] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:00] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:02] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:02] PROBLEM - OCG health on ocg1003 is CRITICAL: CRITICAL: ocg_job_status 312052 msg: ocg_render_job_queue 534 msg (=500 critical) [06:32:03] PROBLEM - OCG health on ocg1001 is CRITICAL: CRITICAL: ocg_job_status 312057 msg: ocg_render_job_queue 509 msg (=500 critical) [06:32:03] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:09] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:10] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:10] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:11] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:11] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 2 failures [06:33:00] RECOVERY - OCG health on ocg1003 is OK: OK: ocg_job_status 312158 msg: ocg_render_job_queue 35 msg [06:33:09] RECOVERY - OCG health on ocg1001 is OK: OK: ocg_job_status 312163 msg: ocg_render_job_queue 29 msg [06:33:30] RECOVERY - OCG health on ocg1002 is OK: OK: ocg_job_status 312208 msg: ocg_render_job_queue 0 msg [06:35:12] PROBLEM - check_puppetrun on barium is CRITICAL: CRITICAL: Puppet has 1 failures [06:37:40] PROBLEM - OCG health on ocg1002 is CRITICAL: CRITICAL: ocg_job_status 315966 msg: ocg_render_job_queue 2010 msg (=500 critical) [06:38:02] dns down O_O [06:38:19] PROBLEM - OCG health on ocg1003 is CRITICAL: CRITICAL: ocg_job_status 317181 msg: ocg_render_job_queue 2884 msg (=500 critical) [06:38:20] PROBLEM - OCG health on ocg1001 is CRITICAL: CRITICAL: ocg_job_status 317230 msg: ocg_render_job_queue 2918 msg (=500 critical) [06:40:09] RECOVERY - check_puppetrun on barium is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:44:50] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:45:24] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:45:39] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:46:00] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:46:01] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:01] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:46:01] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:46:02] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:46:20] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:46:20] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:46:21] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:46:29] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:46:30] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:46:30] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:46:30] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:30] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:46:31] <_joe_> s\: what do you mean with "dns down"? [06:46:39] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:46:42] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:47:00] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:47:00] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:47:01] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:47:01] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:47:05] _joe_ , i can't access commons atm and http://status.wikimedia.org/8777/155942/DNS say performence isuisse [06:47:09] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:47:19] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:47:29] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:47:36] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:47:49] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:48:23] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:50:19] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:37] <_joe_> s\: can I ask you where are you located? [06:50:57] <_joe_> the continent is enough precision :) [06:50:58] Itay, Europe [06:51:10] <_joe_> oh ok, fastweb? [06:51:41] Teletu (=Vodafone) [06:51:50] <_joe_> s\: me too btw, I keep speaking english for everyone else :) [06:52:19] <_joe_> and I access commons just file [06:52:22] <_joe_> *fine [06:55:46] <_joe_> s\: thanks for the report, I'll ping one of the network admins in a few [06:59:19] RECOVERY - OCG health on ocg1003 is OK: OK: ocg_job_status 327625 msg: ocg_render_job_queue 0 msg [06:59:19] RECOVERY - OCG health on ocg1001 is OK: OK: ocg_job_status 327625 msg: ocg_render_job_queue 0 msg [06:59:39] RECOVERY - OCG health on ocg1002 is OK: OK: ocg_job_status 327652 msg: ocg_render_job_queue 0 msg [07:03:18] _joe_: switched to opendns, maybe its only the providers dns O_O [07:03:23] thanks [07:04:54] <_joe_> s\: di niente :) [07:08:54] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:12:42] (03CR) 10Tim Landscheidt: graphite: Add labs archiver script (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/166902 (owner: 10Yuvipanda) [07:22:56] <_joe_> [08:21:07] (03CR) 10PleaseStand: Remove obsolete flags (all of them) from $wgAntiLockFlags [mediawiki-config] - 10https://gerrit.wikimedia.org/r/164012 (owner: 10PleaseStand) [10:28:01] PROBLEM - Router interfaces on cr2-ulsfo is CRITICAL: CRITICAL: host 198.35.26.193, interfaces up: 72, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-1/2/0: down - Core: cr1-eqiad:xe-4/2/1 (Giglinx/Zayo, ETYX/084858//ZYO) {#1062} [10Gbps MPLS]BR [12:25:32] (03PS7) 10KartikMistry: WIP: Update cxserver (beta) config [puppet] - 10https://gerrit.wikimedia.org/r/157787 [13:02:44] (03CR) 10Krinkle: "Remember bug 71761. This removes it from the manifests, but the instances are all still hosting an eager ganglia client daemon trying to t" [puppet] - 10https://gerrit.wikimedia.org/r/165360 (owner: 10Yuvipanda) [13:32:25] (03PS8) 10KartikMistry: WIP: Update cxserver (beta) config [puppet] - 10https://gerrit.wikimedia.org/r/157787 [13:32:48] (03CR) 10KartikMistry: "This should be dropped in favor of, https://gerrit.wikimedia.org/r/#/c/157787/" [puppet] - 10https://gerrit.wikimedia.org/r/163841 (owner: 10KartikMistry) [13:36:49] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: puppet fail [13:36:51] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Puppet has 17 failures [13:36:53] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 52 failures [13:36:59] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 24 failures [13:37:04] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet has 19 failures [13:37:04] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 32 failures [13:37:09] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: puppet fail [13:37:09] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Puppet has 12 failures [13:37:09] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: puppet fail [13:37:09] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 22 failures [13:37:09] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 7 failures [13:37:10] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Puppet has 24 failures [13:37:19] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: puppet fail [13:37:19] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 17 failures [13:37:20] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 65 failures [13:37:20] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 55 failures [13:37:20] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: puppet fail [13:37:20] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 22 failures [13:37:20] PROBLEM - puppetmaster backend https on strontium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [13:37:20] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 9 failures [13:37:21] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 55 failures [13:37:22] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 28 failures [13:37:29] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: puppet fail [13:37:30] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 26 failures [13:37:30] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 18 failures [13:37:30] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: puppet fail [13:37:30] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 19 failures [13:37:30] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 24 failures [13:37:31] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 66 failures [13:37:31] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: puppet fail [13:37:31] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 19 failures [13:37:32] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: puppet fail [13:37:32] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: puppet fail [13:37:36] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 66 failures [13:37:49] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 22 failures [13:37:50] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 42 failures [13:37:50] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Puppet has 65 failures [13:37:50] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 23 failures [13:37:51] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Puppet has 71 failures [13:37:51] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: puppet fail [13:37:51] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 15 failures [13:37:51] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: puppet fail [13:37:52] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 20 failures [13:37:59] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Puppet has 15 failures [13:37:59] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: puppet fail [13:38:00] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 26 failures [13:38:00] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 70 failures [13:38:09] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 14 failures [13:38:10] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Puppet has 19 failures [13:38:10] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 18 failures [13:38:20] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 21 failures [13:38:20] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 19 failures [13:38:20] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Puppet has 20 failures [13:38:20] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 61 failures [13:38:22] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: puppet fail [13:38:29] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Puppet has 20 failures [13:38:33] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: puppet fail [13:38:33] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: puppet fail [13:38:34] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: puppet fail [13:38:35] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: puppet fail [13:38:39] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: puppet fail [13:38:49] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 21 failures [13:38:49] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Puppet has 20 failures [13:38:49] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: puppet fail [13:38:49] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 20 failures [13:38:50] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 66 failures [13:38:50] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: puppet fail [13:38:50] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 45 failures [13:38:50] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Puppet has 83 failures [13:38:51] PROBLEM - puppet last run on es2006 is CRITICAL: CRITICAL: puppet fail [13:38:51] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 21 failures [13:38:52] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 58 failures [13:38:53] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 71 failures [13:39:04] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 20 failures [13:39:06] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 15 failures [13:39:06] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 59 failures [13:39:13] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 62 failures [13:39:13] PROBLEM - puppet last run on cp1008 is CRITICAL: CRITICAL: puppet fail [13:39:13] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 21 failures [13:39:13] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: puppet fail [13:39:20] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 19 failures [13:39:21] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Puppet has 22 failures [13:39:21] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 29 failures [13:39:21] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 57 failures [13:39:29] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 35 failures [13:39:30] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: puppet fail [13:39:30] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: puppet fail [13:39:30] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: puppet fail [13:39:30] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 60 failures [13:39:30] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: puppet fail [13:39:30] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Puppet has 20 failures [13:39:31] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: puppet fail [13:39:31] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Puppet has 20 failures [13:39:32] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 29 failures [13:39:32] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 18 failures [13:39:39] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: puppet fail [13:39:40] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 17 failures [13:39:40] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 64 failures [13:39:40] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 21 failures [13:39:41] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Puppet has 24 failures [13:39:49] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 20 failures [13:39:50] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 77 failures [13:39:50] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 24 failures [13:39:55] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 54 failures [13:39:55] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: puppet fail [13:39:55] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 68 failures [13:39:55] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Puppet has 51 failures [13:39:59] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Puppet has 66 failures [13:40:00] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 56 failures [13:40:00] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: puppet fail [13:40:00] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: puppet fail [13:40:09] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: puppet fail [13:40:10] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: puppet fail [13:40:10] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 62 failures [13:40:10] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: puppet fail [13:40:10] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 17 failures [13:40:11] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 22 failures [13:40:11] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: puppet fail [13:40:19] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: Puppet has 24 failures [13:40:19] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 31 failures [13:40:19] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 64 failures [13:40:19] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: puppet fail [13:40:19] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: puppet fail [13:40:20] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: puppet fail [13:40:20] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 23 failures [13:40:20] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 59 failures [13:40:21] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: puppet fail [13:40:22] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: puppet fail [13:40:29] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 54 failures [13:40:31] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 59 failures [13:40:31] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 26 failures [13:40:39] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: puppet fail [13:40:39] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: puppet fail [13:40:40] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [13:40:40] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 24 failures [13:40:40] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 55 failures [13:40:40] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 18 failures [13:40:40] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 65 failures [13:40:47] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 71 failures [13:40:49] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 19 failures [13:40:59] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 27 failures [13:40:59] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 20 failures [13:41:00] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 24 failures [13:41:00] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: puppet fail [13:41:00] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: puppet fail [13:41:01] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: puppet fail [13:41:01] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: puppet fail [13:41:01] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 59 failures [13:41:01] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 17 failures [13:41:01] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Puppet has 52 failures [13:41:09] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 23 failures [13:41:09] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Puppet has 19 failures [13:41:10] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 65 failures [13:41:19] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 69 failures [13:41:20] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: puppet fail [13:41:20] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 70 failures [13:41:20] PROBLEM - puppet last run on db2033 is CRITICAL: CRITICAL: puppet fail [13:41:20] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 17 failures [13:41:20] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Puppet has 24 failures [13:41:21] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 56 failures [13:41:29] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 22 failures [13:41:33] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet has 26 failures [13:41:35] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 23 failures [13:41:36] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 61 failures [13:41:39] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: puppet fail [13:41:39] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: puppet fail [13:41:40] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: puppet fail [13:41:40] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: puppet fail [13:41:49] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 23 failures [13:41:50] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 22 failures [13:41:59] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 54 failures [13:41:59] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: puppet fail [13:42:00] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 23 failures [13:42:00] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: puppet fail [13:42:00] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 26 failures [13:42:13] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: puppet fail [13:42:15] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: puppet fail [13:42:15] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 77 failures [13:42:16] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: puppet fail [13:42:16] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 25 failures [13:42:16] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 16 failures [13:42:16] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: puppet fail [13:42:16] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Puppet has 16 failures [13:42:17] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 24 failures [13:42:20] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: Puppet has 47 failures [13:42:29] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: puppet fail [13:42:29] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 24 failures [13:42:29] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: puppet fail [13:42:29] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 66 failures [13:42:30] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 77 failures [13:42:30] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Puppet has 17 failures [13:42:30] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: puppet fail [13:42:30] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 25 failures [13:42:31] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 68 failures [13:42:31] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 68 failures [13:42:32] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: puppet fail [13:42:40] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: puppet fail [13:42:43] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: puppet fail [13:42:49] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 28 failures [13:42:49] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 75 failures [13:43:06] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 19 failures [13:43:09] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Puppet has 17 failures [13:43:09] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: puppet fail [13:43:09] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: puppet fail [13:43:10] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: puppet fail [13:43:10] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 25 failures [13:43:11] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 70 failures [13:43:19] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 21 failures [13:43:20] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: puppet fail [13:43:20] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: puppet fail [13:43:20] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Puppet has 22 failures [13:43:21] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 19 failures [13:43:29] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Puppet has 66 failures [13:43:29] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: puppet fail [13:43:29] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 83 failures [13:43:29] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: puppet fail [13:43:30] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: puppet fail [13:43:30] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 30 failures [13:43:30] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 61 failures [13:43:31] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: puppet fail [13:43:31] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: puppet fail [13:43:44] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet has 25 failures [13:43:44] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: puppet fail [13:43:45] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Puppet has 21 failures [13:43:50] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: puppet fail [13:43:50] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: puppet fail [13:43:50] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: puppet fail [13:43:51] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Puppet has 25 failures [13:43:51] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Puppet has 23 failures [13:43:51] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 63 failures [13:44:01] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet has 71 failures [13:44:01] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: puppet fail [13:44:02] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: puppet fail [13:44:03] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: puppet fail [13:44:03] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: puppet fail [13:44:03] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Puppet has 63 failures [13:44:03] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 60 failures [13:44:03] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: puppet fail [13:44:11] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Puppet has 22 failures [13:44:11] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 20 failures [13:44:12] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Puppet has 65 failures [13:44:12] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: puppet fail [13:44:14] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 65 failures [13:44:14] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 14 failures [13:44:14] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: puppet fail [13:44:21] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: puppet fail [13:44:21] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: puppet fail [13:44:21] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 21 failures [13:44:21] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: puppet fail [13:44:21] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: puppet fail [13:44:31] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 61 failures [13:44:31] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: puppet fail [13:44:32] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: puppet fail [13:44:32] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 35 failures [13:44:32] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: puppet fail [13:44:32] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: puppet fail [13:44:32] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: puppet fail [13:44:33] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 21 failures [13:44:33] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: puppet fail [13:44:33] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: puppet fail [13:44:34] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: puppet fail [13:44:35] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 20 failures [13:44:35] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: Puppet has 47 failures [13:44:36] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 67 failures [13:44:36] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 23 failures [13:44:36] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: puppet fail [13:44:41] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: puppet fail [13:44:42] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: puppet fail [13:44:42] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: puppet fail [13:44:42] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: puppet fail [13:44:42] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: puppet fail [13:44:52] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: puppet fail [13:44:53] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: puppet fail [13:44:53] PROBLEM - puppet last run on db1017 is CRITICAL: CRITICAL: puppet fail [13:44:53] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: puppet fail [13:44:53] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: puppet fail [13:44:53] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 70 failures [13:44:54] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Puppet has 59 failures [13:45:01] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: puppet fail [13:45:01] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 23 failures [13:45:01] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 57 failures [13:45:01] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 23 failures [13:45:02] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: puppet fail [13:45:02] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: puppet fail [13:45:12] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: puppet fail [13:45:12] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: puppet fail [13:45:12] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: puppet fail [13:45:21] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 21 failures [13:45:21] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: puppet fail [13:45:21] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Puppet has 16 failures [13:45:22] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 19 failures [13:45:22] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: puppet fail [13:45:22] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: puppet fail [13:45:31] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: puppet fail [13:45:31] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 66 failures [13:45:32] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: puppet fail [13:45:32] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 21 failures [13:45:32] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet has 21 failures [13:45:32] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: puppet fail [13:45:32] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: puppet fail [13:45:32] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: Puppet has 32 failures [13:45:33] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 24 failures [13:45:33] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: puppet fail [13:45:34] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 69 failures [13:45:41] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: puppet fail [13:45:41] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: puppet fail [13:45:41] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 18 failures [13:45:42] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: Puppet has 21 failures [13:45:42] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Puppet has 17 failures [13:45:43] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: puppet fail [13:45:43] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: puppet fail [13:45:43] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet has 61 failures [13:45:43] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: Puppet has 25 failures [13:45:44] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: puppet fail [13:45:44] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: puppet fail [13:45:45] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 22 failures [13:45:51] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: puppet fail [13:45:53] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 18 failures [13:45:53] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: puppet fail [13:45:53] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 64 failures [13:45:53] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: puppet fail [13:45:54] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: puppet fail [13:45:54] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: puppet fail [13:45:54] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 74 failures [13:45:54] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: Puppet has 24 failures [13:46:02] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 20 failures [13:46:02] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 22 failures [13:46:02] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 24 failures [13:46:02] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: puppet fail [13:46:02] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: puppet fail [13:46:11] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: puppet fail [13:46:12] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: puppet fail [13:46:12] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: puppet fail [13:46:12] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 54 failures [13:46:12] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 64 failures [13:46:22] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: puppet fail [13:46:23] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 22 failures [13:46:31] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: Puppet has 21 failures [13:46:32] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 23 failures [13:46:32] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: puppet fail [13:46:32] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet has 57 failures [13:46:32] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 49 failures [13:46:33] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: puppet fail [13:46:33] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 17 failures [13:46:33] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: puppet fail [13:46:33] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 27 failures [13:46:34] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: Puppet has 20 failures [13:46:34] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: puppet fail [13:46:35] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: puppet fail [13:46:41] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 71 failures [13:46:41] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 62 failures [13:46:41] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: puppet fail [13:46:42] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: Puppet has 18 failures [13:46:42] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: puppet fail [13:46:42] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 19 failures [13:46:51] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: puppet fail [13:46:51] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 19 failures [13:46:52] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: puppet fail [13:46:52] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: Puppet has 23 failures [13:46:52] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 64 failures [13:47:01] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 62 failures [13:47:01] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 22 failures [13:47:02] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: puppet fail [13:47:02] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 65 failures [13:47:02] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: puppet fail [13:47:02] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: puppet fail [13:47:02] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: puppet fail [13:47:02] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 58 failures [13:47:11] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: puppet fail [13:47:11] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: puppet fail [13:47:12] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 60 failures [13:47:12] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: puppet fail [13:47:12] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: Puppet has 26 failures [13:47:12] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: puppet fail [13:47:22] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 26 failures [13:47:34] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: puppet fail [13:47:34] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: puppet fail [13:47:34] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 55 failures [13:47:34] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 20 failures [13:47:34] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: puppet fail [13:47:35] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 49 failures [13:47:35] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: puppet fail [13:47:36] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 73 failures [13:47:41] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [13:47:42] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 15 failures [13:47:42] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: puppet fail [13:47:42] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: puppet fail [13:47:43] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: puppet fail [13:47:43] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: Puppet has 22 failures [13:47:44] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: puppet fail [13:47:46] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 21 failures [13:47:46] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: puppet fail [13:47:47] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 67 failures [13:47:51] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: puppet fail [13:47:51] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 22 failures [13:48:01] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: puppet fail [13:48:14] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: puppet fail [13:48:17] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: Puppet has 23 failures [13:48:17] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 61 failures [13:48:18] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 58 failures [13:48:18] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: puppet fail [13:48:18] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: puppet fail [13:48:19] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: puppet fail [13:48:19] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 66 failures [13:48:19] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 22 failures [13:48:28] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 21 failures [13:48:28] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 72 failures [13:48:29] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 69 failures [13:48:29] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: puppet fail [13:48:38] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: puppet fail [13:48:39] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 28 failures [13:48:39] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 20 failures [13:48:39] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 71 failures [13:48:39] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 59 failures [13:48:39] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [13:48:39] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: puppet fail [13:48:39] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 21 failures [13:48:40] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: puppet fail [13:48:40] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 61 failures [13:48:41] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: puppet fail [13:48:41] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 20 failures [13:48:49] <_joe_> 5 [13:48:53] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 18 failures [13:48:53] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 62 failures [13:48:54] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: puppet fail [13:48:54] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 36 failures [13:48:58] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: puppet fail [13:48:58] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: puppet fail [13:48:58] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: puppet fail [13:49:08] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: puppet fail [13:49:08] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: puppet fail [13:49:08] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: puppet fail [13:49:08] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: puppet fail [13:49:08] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 37 failures [13:49:09] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: puppet fail [13:49:09] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: puppet fail [13:49:10] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: puppet fail [13:49:10] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: puppet fail [13:49:10] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: puppet fail [13:49:11] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 53 failures [13:49:11] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: puppet fail [13:49:14] <_joe_> !log restarted apache on the puppetmasters [13:49:18] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Puppet has 23 failures [13:49:19] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 53 failures [13:49:22] Logged the message, Master [13:49:29] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: puppet fail [13:49:29] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 19 failures [13:49:29] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: puppet fail [13:49:29] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: puppet fail [13:49:29] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.994 second response time [13:49:31] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: puppet fail [13:49:33] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: puppet fail [13:49:33] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: puppet fail [13:49:33] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 25 failures [13:49:41] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: puppet fail [13:49:41] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 59 failures [13:49:49] PROBLEM - puppet last run on lithium is CRITICAL: CRITICAL: puppet fail [13:49:49] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 22 failures [13:49:49] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Puppet has 25 failures [13:49:49] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 26 failures [13:49:50] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: puppet fail [13:49:50] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 31 failures [13:49:50] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: puppet fail [13:49:58] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 21 failures [13:49:58] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 57 failures [13:49:59] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: puppet fail [13:49:59] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 68 failures [13:49:59] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 22 failures [13:49:59] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: puppet fail [13:50:08] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 64 failures [13:50:08] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 72 failures [13:50:09] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: puppet fail [13:50:09] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: puppet fail [13:50:09] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Puppet has 14 failures [13:50:09] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 21 failures [13:50:09] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: puppet fail [13:50:09] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [13:50:23] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: puppet fail [13:50:23] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: puppet fail [13:50:23] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: puppet fail [13:50:23] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: puppet fail [13:50:23] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: puppet fail [13:50:23] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: puppet fail [13:50:24] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: Puppet has 20 failures [13:50:24] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 24 failures [13:50:25] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: puppet fail [13:50:26] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: puppet fail [13:50:26] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: puppet fail [13:50:27] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 22 failures [13:50:29] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: puppet fail [13:50:29] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [13:50:30] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: puppet fail [13:50:30] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: puppet fail [13:50:38] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 65 failures [13:50:38] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 16 failures [13:50:39] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: puppet fail [13:50:39] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Puppet has 48 failures [13:50:39] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 80 failures [13:50:39] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: puppet fail [13:50:39] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Puppet has 28 failures [13:50:49] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 18 failures [13:50:49] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: puppet fail [13:50:49] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: puppet fail [13:50:49] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: puppet fail [13:50:49] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 68 failures [13:50:50] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 25 failures [13:50:58] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: puppet fail [13:50:58] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: puppet fail [13:50:59] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:50:59] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 25 failures [13:51:08] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: puppet fail [13:51:11] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: puppet fail [13:51:12] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: puppet fail [13:51:13] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: puppet fail [13:51:13] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: puppet fail [13:51:13] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 54 failures [13:51:13] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: puppet fail [13:51:18] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: puppet fail [13:51:18] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: puppet fail [13:51:18] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 22 failures [13:51:19] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 62 failures [13:51:19] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Puppet has 7 failures [13:51:19] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: Puppet has 26 failures [13:51:19] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: puppet fail [13:51:19] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 64 failures [13:51:20] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Puppet has 28 failures [13:51:20] PROBLEM - puppet last run on plutonium is CRITICAL: CRITICAL: Puppet has 22 failures [13:51:21] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: puppet fail [13:51:21] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: puppet fail [13:51:22] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: puppet fail [13:51:28] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: puppet fail [13:51:29] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: puppet fail [13:51:29] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: puppet fail [13:51:29] PROBLEM - puppet last run on db2001 is CRITICAL: CRITICAL: puppet fail [13:51:29] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Puppet has 23 failures [13:51:29] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 61 failures [13:51:29] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 20 failures [13:51:30] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Puppet has 23 failures [13:51:39] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Puppet has 65 failures [13:51:39] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Puppet has 17 failures [13:51:48] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 22 failures [13:51:56] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Puppet has 21 failures [13:51:56] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Puppet has 24 failures [13:52:08] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: Puppet has 5 failures [13:54:09] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:54:19] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:54:28] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:54:29] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [13:54:48] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:54:52] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:55:08] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [13:55:09] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:55:18] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:55:19] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:55:19] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:55:29] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:55:29] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:55:29] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [13:55:29] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:55:30] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:55:40] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:55:40] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:55:48] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [13:55:49] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:55:49] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:55:49] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:55:49] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:55:50] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:55:50] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:55:51] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:55:51] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:55:51] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:55:59] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:55:59] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:56:08] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:56:08] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:56:09] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:56:09] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:56:15] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:56:17] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:56:17] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:56:17] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:56:28] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:56:28] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:56:29] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [13:56:29] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [13:56:29] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [13:56:29] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:56:38] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:56:38] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:56:38] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:56:39] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:56:39] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:56:39] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:56:39] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:56:39] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:56:48] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:56:49] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:56:50] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:56:58] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:56:58] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [13:56:58] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:56:59] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:56:59] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:56:59] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:57:00] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:57:09] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:57:18] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:57:19] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 72 seconds ago with 0 failures [13:57:19] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:57:19] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:57:19] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:57:19] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:57:19] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 69 seconds ago with 0 failures [13:57:20] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:57:28] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [13:57:28] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:57:28] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:57:30] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:57:38] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:57:41] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:57:44] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:57:52] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:57:52] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:57:52] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:57:59] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:57:59] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:57:59] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:58:00] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [13:58:00] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:58:00] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:58:00] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [13:58:01] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 66 seconds ago with 0 failures [13:58:01] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:58:08] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 72 seconds ago with 0 failures [13:58:09] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:58:09] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:58:09] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:58:09] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:58:10] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:58:18] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:58:18] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:58:18] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:58:18] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:58:18] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:58:19] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:58:19] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:58:19] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:58:20] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [13:58:20] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:58:21] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:58:21] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:58:22] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:58:38] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:58:38] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:58:38] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [13:58:38] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:58:39] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [13:58:39] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:58:39] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:58:39] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:58:40] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:58:40] RECOVERY - puppet last run on cp1008 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [13:58:41] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [13:58:41] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:58:42] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:58:42] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:58:43] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:58:44] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [13:58:44] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:58:48] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:58:48] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:58:48] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:58:48] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:58:49] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [13:58:49] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [13:58:49] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [13:58:58] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:58:58] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:58:59] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:58:59] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [13:58:59] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:58:59] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:58:59] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 63 seconds ago with 0 failures [13:59:00] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [13:59:00] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:59:01] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [13:59:08] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:59:18] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [13:59:23] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:59:23] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [13:59:23] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:59:23] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [13:59:29] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:59:29] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:59:30] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 69 seconds ago with 0 failures [13:59:30] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [13:59:38] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:59:38] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 66 seconds ago with 0 failures [13:59:39] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [13:59:39] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [13:59:39] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:59:39] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [13:59:39] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [13:59:39] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [13:59:40] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:59:40] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:59:48] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [13:59:48] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 71 seconds ago with 0 failures [13:59:48] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:59:48] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:59:49] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:59:49] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:59:51] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:59:54] RECOVERY - puppet last run on db2033 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:59:59] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:00:19] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:00:19] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [14:00:19] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 69 seconds ago with 0 failures [14:00:29] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:00:29] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [14:00:30] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:00:30] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [14:00:38] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [14:00:41] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [14:00:41] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:00:41] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:00:41] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:00:41] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:00:42] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:00:50] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:00:50] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:00:50] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:00:51] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [14:00:51] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:00:51] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [14:00:51] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:00:51] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [14:00:59] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:01:00] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [14:01:01] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 63 seconds ago with 0 failures [14:01:01] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [14:01:01] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:01:09] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:01:10] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:01:10] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:01:10] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:01:19] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [14:01:19] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:01:20] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:01:21] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [14:01:29] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:01:29] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [14:01:38] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:01:42] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [14:01:42] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:01:42] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:01:42] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:01:42] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [14:01:49] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:01:49] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:01:49] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [14:01:49] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:01:49] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 63 seconds ago with 0 failures [14:01:49] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:01:50] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [14:01:50] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [14:01:51] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:01:58] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [14:01:58] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:01:58] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:01:59] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 64 seconds ago with 0 failures [14:01:59] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:01:59] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [14:01:59] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:02:08] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [14:02:08] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:02:09] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:02:09] RECOVERY - puppet last run on db1017 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:02:09] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [14:02:18] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:02:30] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:02:31] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:02:31] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [14:02:31] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [14:02:31] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:02:32] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:02:32] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [14:02:32] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:02:33] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:02:33] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:02:34] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:02:34] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [14:02:34] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:02:38] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [14:02:39] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [14:02:39] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:02:39] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:02:39] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:02:48] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [14:02:48] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:02:49] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [14:02:49] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [14:02:49] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [14:02:49] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:02:49] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [14:02:50] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:02:50] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:02:51] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [14:02:51] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [14:02:51] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [14:02:59] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:03:01] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:03:01] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:03:01] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:03:02] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:03:02] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:03:02] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [14:03:02] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:03:03] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [14:03:08] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:03:13] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:03:13] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [14:03:13] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:03:13] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:03:13] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:03:13] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:03:18] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:03:18] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:03:18] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:03:18] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [14:03:18] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [14:03:18] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [14:03:19] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:03:19] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:03:20] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:03:20] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [14:03:21] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [14:03:21] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:03:29] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:03:29] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:03:29] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:03:38] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [14:03:39] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [14:03:39] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:03:39] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [14:03:39] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:03:40] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:03:48] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [14:03:51] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:03:51] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:03:51] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [14:03:51] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:04:01] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:04:01] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [14:04:02] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [14:04:02] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:04:09] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:04:09] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:04:09] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:04:09] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:04:10] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [14:04:19] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [14:04:22] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:04:22] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 67 seconds ago with 0 failures [14:04:22] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:04:22] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [14:04:28] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [14:04:33] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:04:34] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:04:34] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:04:34] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 64 seconds ago with 0 failures [14:04:34] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [14:04:34] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [14:04:41] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [14:04:41] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [14:04:41] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [14:04:48] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [14:04:48] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [14:04:49] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:04:49] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:04:50] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [14:04:50] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:04:58] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 70 seconds ago with 0 failures [14:04:59] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [14:04:59] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:04:59] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [14:04:59] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:04:59] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:04:59] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [14:05:00] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [14:05:01] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:05:01] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:05:08] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [14:05:08] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:05:09] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [14:05:09] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:05:09] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 66 seconds ago with 0 failures [14:05:09] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:05:09] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [14:05:10] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:05:10] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [14:05:10] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [14:05:11] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:05:11] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [14:05:12] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [14:05:12] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [14:05:13] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [14:05:18] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [14:05:18] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:05:19] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [14:05:19] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [14:05:19] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:05:19] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:05:19] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:05:30] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [14:05:30] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [14:05:31] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [14:05:38] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [14:05:38] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [14:05:38] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:05:38] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [14:05:39] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:05:39] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [14:05:39] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [14:05:39] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [14:05:40] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:05:40] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:05:49] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:05:58] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:05:58] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:05:59] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:05:59] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:06:00] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:06:00] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:06:08] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:06:18] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:06:20] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [14:06:20] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [14:06:21] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:06:22] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [14:06:22] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [14:06:22] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:06:22] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:06:28] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:06:29] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:06:29] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 70 seconds ago with 0 failures [14:06:29] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:06:29] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 63 seconds ago with 0 failures [14:06:30] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [14:06:38] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:06:38] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:06:39] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:06:48] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:06:50] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:06:50] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:06:50] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:06:50] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:06:50] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [14:06:50] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:07:00] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [14:07:00] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [14:07:00] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [14:07:00] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:07:00] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [14:07:08] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:07:09] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:07:09] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:07:09] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:07:09] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [14:07:09] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:07:10] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [14:07:10] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:07:10] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [14:07:18] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [14:07:18] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:07:18] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:07:19] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [14:07:19] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [14:07:19] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:07:19] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:07:28] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [14:07:28] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:07:28] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:07:28] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:07:29] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [14:07:29] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [14:07:38] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [14:07:44] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:07:47] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [14:07:49] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [14:07:49] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:07:49] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [14:07:59] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [14:07:59] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [14:07:59] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:07:59] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [14:08:00] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [14:08:00] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:08:00] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:08:01] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:08:01] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:08:01] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [14:08:02] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [14:08:02] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:08:03] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:08:04] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:08:09] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:08:09] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 67 seconds ago with 0 failures [14:08:09] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:08:10] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:08:10] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [14:08:10] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [14:08:11] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:08:11] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:08:11] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [14:08:19] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:08:20] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:08:20] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:08:20] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [14:08:31] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [14:08:31] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [14:08:31] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [14:08:31] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [14:08:32] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [14:08:41] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [14:08:41] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 64 seconds ago with 0 failures [14:08:42] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:08:42] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:08:42] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [14:08:42] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [14:08:42] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:08:42] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:08:43] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:08:44] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [14:08:44] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 69 seconds ago with 0 failures [14:08:51] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:08:51] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:08:51] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 71 seconds ago with 0 failures [14:08:51] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [14:08:52] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:08:52] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [14:08:53] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [14:08:53] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [14:08:53] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:08:54] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [14:09:01] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:09:03] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [14:09:03] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:09:03] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:09:03] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:09:04] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:09:04] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [14:09:04] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 64 seconds ago with 0 failures [14:09:26] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [14:09:26] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:09:26] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [14:09:30] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:09:31] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [14:09:31] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [14:09:41] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:09:41] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:09:41] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:09:42] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:09:42] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 68 seconds ago with 0 failures [14:09:42] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:09:42] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [14:09:51] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:09:51] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 66 seconds ago with 0 failures [14:09:52] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 67 seconds ago with 0 failures [14:09:52] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [14:09:52] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:10:01] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [14:10:02] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:10:02] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:10:02] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [14:10:02] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:10:03] RECOVERY - puppet last run on db2001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:10:21] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [14:10:44] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [14:11:01] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [14:19:48] (03PS11) 10KartikMistry: Apertium service configuration for Beta [puppet] - 10https://gerrit.wikimedia.org/r/165485 [14:21:12] (03PS12) 10KartikMistry: Beta: Apertium service configuration [puppet] - 10https://gerrit.wikimedia.org/r/165485 [14:23:03] PROBLEM - OCG health on ocg1001 is CRITICAL: CRITICAL: ocg_job_status 340802 msg: ocg_render_job_queue 2272 msg (=500 critical) [14:23:22] PROBLEM - OCG health on ocg1002 is CRITICAL: CRITICAL: ocg_job_status 341414 msg: ocg_render_job_queue 2693 msg (=500 critical) [14:23:33] PROBLEM - OCG health on ocg1003 is CRITICAL: CRITICAL: ocg_job_status 341770 msg: ocg_render_job_queue 2937 msg (=500 critical) [14:25:06] (03PS9) 10KartikMistry: WIP: Beta: Update cxserver to use Apertium service [puppet] - 10https://gerrit.wikimedia.org/r/157787 [14:27:19] (03PS1) 10Giuseppe Lavagetto: Adding the complete memcached fix [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167381 [14:36:17] Request: POST http://en.wikipedia.org/w/index.php?title=User_talk:79.79.161.203&action=submit, from 10.64.32.105 via cp1066 cp1066 ([10.64.0.103]:3128), Varnish XID 2768499007 [14:36:18] Forwarded for: 96.27.83.199, 208.80.154.75, 10.64.32.105 [14:36:18] Error: 503, Service Unavailable at Sat, 18 Oct 2014 14:35:49 GMT [14:42:14] RD: HHVM enabled? [14:42:20] No [14:43:29] <_joe_> "This user is currently blocked. The latest block log entry is provided below for reference" [14:43:37] <_joe_> can't this matter? [14:44:03] Not at all [14:44:05] .. [14:44:12] _joe_: No, he's not the IP [14:44:20] RD: No further details on that page? [14:44:27] * hoo bets on segfault [14:44:34] Nope unfortunately - I got just the one error but the second load was fine [14:44:45] I always drop a note here too tho [14:44:59] <_joe_> RD: the second was fine? so you were able to edit the page after the first attempt? [14:45:07] Clearly. [14:45:15] You just looked at the page :) [14:45:34] <_joe_> hoo: then it's not a segfault, I bet [14:45:45] <_joe_> but some temporary hiccup of some backend [14:45:57] * RD nods [14:46:21] But yeah - I just drop a note here anyways in case anybody happened to be tinkering ;-) [14:46:33] <_joe_> yes thanks a lot! [14:46:41] <_joe_> I am actually taking a look [14:47:28] <_joe_> ls [14:47:47] <_joe_> the cat moved the mouse with her tail... [14:48:16] can't see anything obvious in either exception nor fatal log [14:48:24] <_joe_> me neither [14:48:38] <_joe_> hoo: I see mw1173 is quite overloaded, for instance [14:48:43] <_joe_> that may be enough [14:48:50] <_joe_> I'll take a look in a few minutes [14:48:56] * _joe_ needs coffee [14:49:44] * FlorianSW gives _joe_ a cup of coffee [14:49:46] It's indeed quite loaded, but not to the point where stuff goes randomly crazy [14:49:55] dmesg looks ok-ish [14:50:17] so stuff should at least "properly" time out on it [14:50:28] properly as in with a log entry in fatal.log [14:52:33] RECOVERY - OCG health on ocg1002 is OK: OK: ocg_job_status 351451 msg: ocg_render_job_queue 74 msg [14:52:51] RECOVERY - OCG health on ocg1003 is OK: OK: ocg_job_status 351488 msg: ocg_render_job_queue 0 msg [14:53:12] RECOVERY - OCG health on ocg1001 is OK: OK: ocg_job_status 351519 msg: ocg_render_job_queue 0 msg [15:07:05] (03PS1) 10Springle: repool db1061 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/167382 [15:08:04] (03CR) 10Springle: [C: 032] repool db1061 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/167382 (owner: 10Springle) [15:08:10] (03Merged) 10jenkins-bot: repool db1061 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/167382 (owner: 10Springle) [15:08:48] PROBLEM - OCG health on ocg1001 is CRITICAL: CRITICAL: ocg_job_status 354975 msg: ocg_render_job_queue 1103 msg (=500 critical) [15:09:18] PROBLEM - OCG health on ocg1002 is CRITICAL: CRITICAL: ocg_job_status 355334 msg: ocg_render_job_queue 1319 msg (=500 critical) [15:09:20] !log springle Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 06s) [15:09:26] Logged the message, Master [15:09:37] PROBLEM - OCG health on ocg1003 is CRITICAL: CRITICAL: ocg_job_status 355699 msg: ocg_render_job_queue 1533 msg (=500 critical) [15:22:19] (03PS1) 10Giuseppe Lavagetto: Workaround dpkg-source barfing on valid udiff [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167383 [15:22:21] (03PS1) 10Giuseppe Lavagetto: Disable pretty CMAKE build [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167384 [15:22:23] (03PS1) 10Giuseppe Lavagetto: Fix hhvm-dbg to actually include debugging symbols [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167385 [15:22:25] (03PS1) 10Giuseppe Lavagetto: Sync RelWithDebInfo patch with 3.3.0 [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167386 [15:22:55] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Adding the complete memcached fix [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167381 (owner: 10Giuseppe Lavagetto) [15:23:15] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Fix hhvm-dbg to actually include debugging symbols [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167385 (owner: 10Giuseppe Lavagetto) [15:23:38] (03CR) 10Giuseppe Lavagetto: [C: 032] Workaround dpkg-source barfing on valid udiff [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167383 (owner: 10Giuseppe Lavagetto) [15:23:46] (03CR) 10Giuseppe Lavagetto: [V: 032] Workaround dpkg-source barfing on valid udiff [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167383 (owner: 10Giuseppe Lavagetto) [15:24:08] <_joe_> (almost all of this is faidon's work, not mine) [15:24:16] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Disable pretty CMAKE build [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167384 (owner: 10Giuseppe Lavagetto) [15:25:28] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Sync RelWithDebInfo patch with 3.3.0 [debs/hhvm] - 10https://gerrit.wikimedia.org/r/167386 (owner: 10Giuseppe Lavagetto) [16:03:28] RECOVERY - OCG health on ocg1002 is OK: OK: ocg_job_status 333303 msg: ocg_render_job_queue 33 msg [16:03:48] RECOVERY - OCG health on ocg1003 is OK: OK: ocg_job_status 333318 msg: ocg_render_job_queue 0 msg [16:04:07] RECOVERY - OCG health on ocg1001 is OK: OK: ocg_job_status 333338 msg: ocg_render_job_queue 0 msg [16:46:16] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Added initial Debian packaging [debs/contenttranslation/apertium-apy] - 10https://gerrit.wikimedia.org/r/166393 (owner: 10KartikMistry) [17:06:31] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 1 failures [17:24:13] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [17:37:16] (03PS1) 10BryanDavis: iegreview: Enable rewrite engine in Location context [puppet] - 10https://gerrit.wikimedia.org/r/167394 (https://bugzilla.wikimedia.org/72201) [18:16:14] PROBLEM - OCG health on ocg1003 is CRITICAL: CRITICAL: ocg_job_status 348830 msg: ocg_render_job_queue 2544 msg (=500 critical) [18:16:23] PROBLEM - OCG health on ocg1001 is CRITICAL: CRITICAL: ocg_job_status 349300 msg: ocg_render_job_queue 2908 msg (=500 critical) [18:16:54] PROBLEM - OCG health on ocg1002 is CRITICAL: CRITICAL: ocg_job_status 349904 msg: ocg_render_job_queue 3393 msg (=500 critical) [18:50:31] RECOVERY - OCG health on ocg1001 is OK: OK: ocg_job_status 357009 msg: ocg_render_job_queue 95 msg [18:50:59] RECOVERY - OCG health on ocg1002 is OK: OK: ocg_job_status 357046 msg: ocg_render_job_queue 39 msg [18:51:10] RECOVERY - OCG health on ocg1003 is OK: OK: ocg_job_status 357068 msg: ocg_render_job_queue 0 msg [21:49:44] (03PS1) 10Legoktm: Set 'extensionListUrl' setting in ExtensionDistributor config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/167408 [22:05:52] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [22:12:52] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: puppet fail [22:26:32] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [22:31:33] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [22:58:16] (03PS1) 10Ori.livneh: mathoid: cleanup [puppet] - 10https://gerrit.wikimedia.org/r/167413 [23:01:43] (03PS2) 10Ori.livneh: mathoid: cleanup [puppet] - 10https://gerrit.wikimedia.org/r/167413 [23:13:50] * AndyRussG waves at ori [23:14:04] Hi ori... may I bug you with a quick question? It is: where would I go to find out how DB transations work in batch jobs and maintenance scripts? I want to make a job and a script that add and remove a million or so rows for fun and profit [23:46:48] AndyRussG: A million inserts in a single transaction? That's not a good idea (unless your alone on a server... then it might be ok-ish) [23:47:11] hoo: exactly [23:47:36] AndyRussG: So... can you split the transaction? [23:47:51] yes that's what I want to do :) [23:47:53] https://www.mediawiki.org/wiki/Manual:Database_access#Lock_contention [23:48:56] My understanding is that when running in a normal Web request, there is some automatic handling of locking and transactions [23:49:27] So I'm just trying to figure out how things may be different in maintenance scripts and batch jobs [23:49:47] AndyRussG: If you want a transaction there, you have to open one [23:50:09] * begin [23:50:11] So all I need to do is call $dbw->begin() and $dbw->commit() around every write operation? [23:50:22] if you want them in a transaction, yes [23:50:37] if you don't care for transactions, just doing your queries should be fine [23:50:42] Basically I don't care about it taking a long time (since it's batch/script mode) but I don't want to lock up master [23:52:41] I'm not much worried about consistency, this corner of the site doesn't get such heavy use [23:53:30] AndyRussG: In that case, just do your inserts (up to 5k or so max at a time), wait for slaves... profit [23:54:56] hoo: here's a very-WIP fragment in case you're interested: https://gerrit.wikimedia.org/r/#/c/167243/2/includes/Campaign.php [23:55:50] There's a single delete call that could take out a few million rows in one swoop, line 1132 [23:56:11] AndyRussG: Not a good idea :P [23:56:18] Make that a loop and wait for slaves [23:56:23] Hmmm.... [23:56:44] So put some kind of limit on it to make it do a smaller number at a time? [23:56:58] yep, do it in small batches [23:57:24] The inserts go one row at a time, so I could put a counter in there and call waitforslaves every now and again, line 1192 [23:57:30] because you don't want all slaves being busy clearing out 2 million rows and not replicating anything else [23:57:40] (replication is synchronous) [23:57:58] Ah hmmm [23:58:00] OK! [23:58:22] AndyRussG: If you really want to do multiple millions you probably want to give an array with multiple rows to insert [23:58:27] but not all of them at once, ofc [23:58:34] And it's certain that writes to master won't be tied up for the duration of my script/job's run? [23:58:41] It could last many minutes [23:59:09] That was my initial concern--thanks 4 pointing out the replication issue, I hadn't thought of that 8p [23:59:45] well, usually the slaves are slower with writes, so master wont get hammered down because you are waiting for slaves