[00:31:45] (03CR) 10Hoo man: [C: 031] "Trivial... ping me in case this doesn't go through before the 31st" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181791 (owner: 10Nemo bis) [02:33:23] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: puppet fail [02:50:23] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:27:52] PROBLEM - RAID on virt1003 is CRITICAL: CRITICAL: Active: 14, Working: 14, Failed: 2, Spare: 0 [03:34:22] RECOVERY - RAID on virt1003 is OK: OK: Active: 14, Working: 14, Failed: 0, Spare: 0 [03:52:12] PROBLEM - Disk space on cerium is CRITICAL: DISK CRITICAL - free space: /mnt/data 10891 MB (3% inode=99%): [03:59:03] PROBLEM - Disk space on cerium is CRITICAL: DISK CRITICAL - free space: /mnt/data 11093 MB (3% inode=99%): [04:59:32] PROBLEM - Disk space on cerium is CRITICAL: DISK CRITICAL - free space: /mnt/data 10766 MB (3% inode=99%): [05:23:57] PROBLEM - RAID on ms-be2011 is CRITICAL: CRITICAL: 1 failed LD(s) (Offline) [05:25:27] PROBLEM - Disk space on ms-be2011 is CRITICAL: DISK CRITICAL - /srv/swift-storage/sda1 is not accessible: Input/output error [05:26:15] PROBLEM - Disk space on cerium is CRITICAL: DISK CRITICAL - free space: /mnt/data 11330 MB (3% inode=99%): [05:28:17] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 3.60736443136e-37 [05:34:26] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 1 failures [05:42:56] PROBLEM - Disk space on xenon is CRITICAL: DISK CRITICAL - free space: /mnt/data 11020 MB (3% inode=99%): [05:43:20] gwicke: cassandra on cerium is running out of space [05:49:48] !log cerium disk space critical, so moved /mnt/data/cassandra/java_{1418354329,1418533386,1418537719}.hprof to /tmp/hprof_files, freeing up ~17G of space. [05:49:57] Logged the message, Master [05:49:59] RECOVERY - Disk space on cerium is OK: DISK OK [06:02:37] RECOVERY - Disk space on ms-be2011 is OK: DISK OK [06:34:51] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:56] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:57] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:06] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:17] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 4 failures [06:35:18] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:36:16] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:30] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:31] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 1 failures [06:37:06] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:37:28] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [06:38:22] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:40:00] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:40:02] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:27] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:45:27] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:45:38] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:45:56] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:45:57] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:46:37] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:46:49] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:47:39] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:57] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:48:47] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:48:57] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:50:23] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:50:23] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:50:23] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:18:47] PROBLEM - Disk space on praseodymium is CRITICAL: DISK CRITICAL - free space: /mnt/data 11128 MB (3% inode=99%): [07:29:29] PROBLEM - Disk space on cerium is CRITICAL: DISK CRITICAL - free space: /mnt/data 11130 MB (3% inode=99%): [07:56:34] RECOVERY - Disk space on cerium is OK: DISK OK [08:45:23] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 7.136016132e-43 [09:45:27] PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 633 [09:50:26] RECOVERY - check_mysql on db1008 is OK: Uptime: 6294206 Threads: 5 Questions: 190784521 Slow queries: 44158 Opens: 119250 Flush tables: 2 Open tables: 64 Queries per second avg: 30.311 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [10:28:46] (03CR) 10Lydia Pintscher: "Hey :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181871 (owner: 10Tpt) [10:39:04] PROBLEM - Host logstash1003 is DOWN: CRITICAL - Plugin timed out after 15 seconds [10:39:04] PROBLEM - Host dbproxy1001 is DOWN: CRITICAL - Plugin timed out after 15 seconds [10:39:04] PROBLEM - Host elastic1003 is DOWN: CRITICAL - Plugin timed out after 15 seconds [10:39:04] PROBLEM - Host db2029 is DOWN: CRITICAL - Plugin timed out after 15 seconds [10:39:04] PROBLEM - Host db2018 is DOWN: CRITICAL - Plugin timed out after 15 seconds [10:39:14] RECOVERY - Host logstash1003 is UP: PING OK - Packet loss = 0%, RTA = 1.07 ms [10:39:14] RECOVERY - Host db2029 is UP: PING OK - Packet loss = 0%, RTA = 43.01 ms [10:39:15] RECOVERY - Host elastic1003 is UP: PING OK - Packet loss = 0%, RTA = 3.36 ms [10:39:25] RECOVERY - Host dbproxy1001 is UP: PING OK - Packet loss = 0%, RTA = 3.15 ms [10:39:25] RECOVERY - Host db2018 is UP: PING OK - Packet loss = 0%, RTA = 43.83 ms [10:47:47] (03CR) 10Hoo man: "The toolbox link is not going to be affected by this at all." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181871 (owner: 10Tpt) [10:53:03] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: puppet fail [10:55:03] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [11:06:56] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:08:46] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [12:08:37] (03PS1) 10Springle: repool db1066 and db1028 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181884 [12:10:15] (03CR) 10Springle: [C: 032] repool db1066 and db1028 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181884 (owner: 10Springle) [12:10:19] (03Merged) 10jenkins-bot: repool db1066 and db1028 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181884 (owner: 10Springle) [12:11:27] !log springle Synchronized wmf-config/db-eqiad.php: repool db1066 and db1028, warm up (duration: 00m 06s) [12:11:30] Logged the message, Master [12:39:29] (03PS1) 10Springle: bump up s1 api load sent to db1066 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181885 [12:53:43] (03CR) 10Springle: [C: 032 V: 032] bump up s1 api load sent to db1066 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181885 (owner: 10Springle) [12:55:09] !log springle Synchronized wmf-config/db-eqiad.php: bump up s1 api load sent to db1066 (duration: 00m 06s) [12:55:11] Logged the message, Master [13:32:08] PROBLEM - Host labs-ns1.wikimedia.org is DOWN: CRITICAL - Plugin timed out after 15 seconds [13:32:08] PROBLEM - Host wikidata is DOWN: CRITICAL - Plugin timed out after 15 seconds [13:35:45] RECOVERY - Host labs-ns1.wikimedia.org is UP: PING OK - Packet loss = 0%, RTA = 43.46 ms [13:39:39] RECOVERY - Host wikidata is UP: PING OK - Packet loss = 0%, RTA = 95.74 ms [14:11:28] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: puppet fail [14:16:11] jenkins seems to be down. anyone aware of that? [14:23:03] interesting: while jenkins goes down on gallium, the load there increases. https://ganglia.wikimedia.org/latest/?r=4hr&cs=&ce=&m=cpu_report&tab=ch&vn=&hide-hf=false&hreg[]=gallium [14:28:24] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:38:33] (03PS2) 10Lydia Pintscher: Display links to Wikidata in the other project sidebar [mediawiki-config] - 10https://gerrit.wikimedia.org/r/181871 (owner: 10Tpt) [15:21:54] (03PS1) 10Reedy: Add wikimania2016 [dns] - 10https://gerrit.wikimedia.org/r/181891 [15:22:44] ooh [15:23:36] It's been announced, they're gonna want a wiki soon.. ;) [15:26:06] (03PS1) 10Reedy: Add wikimania2016.wikimedia.org to ServerAlias [puppet] - 10https://gerrit.wikimedia.org/r/181892 [15:33:18] Reedy: announced where? I'm not sure it's public yet [15:33:36] Though there were press releases already on the 24th :p [15:34:34] (03PS2) 10Reedy: Add wikimania2016 [dns] - 10https://gerrit.wikimedia.org/r/181891 [15:34:42] (03PS2) 10Reedy: Add wikimania2016.wikimedia.org to ServerAlias [puppet] - 10https://gerrit.wikimedia.org/r/181892 [15:35:03] Nemo_bis: I've seen people tweeting/posting it on FB... [15:35:13] Though, I've not outed it ;) [15:36:06] Yes, it's not a well kept secret ^^ [16:11:15] !log jenkins is hung with high cpu/memory usage [16:11:19] Logged the message, Master [16:12:00] !log attempting to kill jenkins [16:12:03] Logged the message, Master [16:15:10] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49758 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311642 MB (80% inode=99%): [16:17:55] !log jenkins killed [16:17:59] Logged the message, Master [16:18:59] !log jenkins started again... [16:19:02] Logged the message, Master [16:20:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49757 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311629 MB (80% inode=99%): [16:25:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49757 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311616 MB (80% inode=99%): [16:27:48] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 0 below the confidence bounds [16:30:13] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49757 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311603 MB (80% inode=99%): [16:35:14] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49757 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311590 MB (80% inode=99%): [16:40:13] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49757 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311577 MB (80% inode=99%): [16:45:09] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49757 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311563 MB (80% inode=99%): [16:50:20] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49757 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311550 MB (80% inode=99%): [16:55:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49757 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311537 MB (80% inode=99%): [17:00:15] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49756 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311524 MB (80% inode=99%): [17:03:19] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 1 failures [17:05:11] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49756 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311511 MB (80% inode=99%): [17:10:13] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49756 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311498 MB (80% inode=99%): [17:15:07] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49756 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311485 MB (80% inode=99%): [17:17:27] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:20:09] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49756 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311472 MB (80% inode=99%): [17:25:07] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49756 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311458 MB (80% inode=99%): [17:30:22] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49756 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311445 MB (80% inode=99%): [17:35:04] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49756 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311432 MB (80% inode=99%): [17:40:19] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49755 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311419 MB (80% inode=99%): [17:45:09] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49755 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311405 MB (80% inode=99%): [17:49:38] ^Silicon is a frack machine [17:50:08] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49755 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311392 MB (80% inode=99%): [17:55:16] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49755 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311379 MB (80% inode=99%): [18:00:16] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49755 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311366 MB (80% inode=99%): [18:05:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49755 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311353 MB (80% inode=99%): [18:07:43] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [18:10:14] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49755 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311340 MB (80% inode=99%): [18:14:34] RECOVERY - Disk space on praseodymium is OK: DISK OK [18:15:14] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49755 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311326 MB (80% inode=99%): [18:16:37] (03CR) 10MZMcBride: [C: 031] Grant reedy root [puppet] - 10https://gerrit.wikimedia.org/r/122621 (owner: 10Reedy) [18:19:13] :o [18:20:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49754 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311313 MB (80% inode=99%): [18:25:13] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49754 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311300 MB (80% inode=99%): [18:30:15] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49754 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311287 MB (80% inode=99%): [18:35:15] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49754 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311273 MB (80% inode=99%): [18:40:18] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49754 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311260 MB (80% inode=99%): [18:43:06] PROBLEM - Disk space on xenon is CRITICAL: DISK CRITICAL - free space: /mnt/data 11042 MB (3% inode=99%): [18:55:16] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49754 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311220 MB (80% inode=99%): [18:56:26] RECOVERY - Disk space on xenon is OK: DISK OK [19:00:18] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49753 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311207 MB (80% inode=99%): [19:03:26] ori, thanks for moving those files; do you know where they come from? [19:05:18] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49753 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311194 MB (80% inode=99%): [19:10:15] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49753 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311181 MB (80% inode=99%): [19:15:13] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49753 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311168 MB (80% inode=99%): [19:20:09] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49753 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311154 MB (80% inode=99%): [19:25:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49753 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311141 MB (80% inode=99%): [19:30:11] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49753 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311128 MB (80% inode=99%): [19:45:14] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49752 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311088 MB (80% inode=99%): [19:50:19] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49752 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311075 MB (80% inode=99%): [19:53:38] !log gallium: restarted jenkins [19:53:43] Logged the message, Master [19:53:46] ori: again? [19:53:54] lots of cpu and memory? [19:54:32] yep [19:55:11] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49752 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311062 MB (80% inode=99%): [20:00:06] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49752 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311048 MB (80% inode=99%): [20:05:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49752 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311035 MB (80% inode=99%): [20:10:08] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49752 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311022 MB (80% inode=99%): [20:15:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49752 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 311009 MB (80% inode=99%): [20:20:09] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49751 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310995 MB (80% inode=99%): [20:25:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49751 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310982 MB (80% inode=99%): [20:26:12] PROBLEM - Disk space on xenon is CRITICAL: DISK CRITICAL - free space: /mnt/data 10445 MB (3% inode=99%): [20:29:22] RECOVERY - Disk space on xenon is OK: DISK OK [20:29:33] !log dropped old keyspaces titan{,2,3} on xenon to free space for titan4 [20:29:40] Logged the message, Master [20:30:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49751 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310969 MB (80% inode=99%): [20:35:11] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49751 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310956 MB (80% inode=99%): [20:40:09] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49751 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310942 MB (80% inode=99%): [20:45:14] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49751 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310929 MB (80% inode=99%): [20:46:12] PROBLEM - Host search-pool1.svc.eqiad.wmnet is DOWN: CRITICAL - Plugin timed out after 15 seconds [20:47:50] hmm [20:49:40] RECOVERY - Host search-pool1.svc.eqiad.wmnet is UP: PING OK - Packet loss = 0%, RTA = 1.03 ms [20:50:24] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49751 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310916 MB (80% inode=99%): [20:51:07] (i did nothing, search recovered on its own) [20:51:35] jgage: you're magic :D [20:51:47] i gave it a stern look ;) [20:55:17] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49751 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310903 MB (80% inode=99%): [21:00:09] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49750 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310889 MB (80% inode=99%): [21:05:19] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49750 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310876 MB (80% inode=99%): [21:10:07] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49750 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310863 MB (80% inode=99%): [21:14:36] hey, so search-pool ? [21:15:10] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49750 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310850 MB (80% inode=99%): [21:20:12] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49750 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310837 MB (80% inode=99%): [21:25:14] PROBLEM - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49750 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310824 MB (80% inode=99%): [21:29:38] ACKNOWLEDGEMENT - check_disk on silicon is CRITICAL: DISK CRITICAL - free space: / 49750 MB (93% inode=97%): /boot 180 MB (68% inode=99%): /a 310824 MB (80% inode=99%): alexandros kosiaris silencing, informed Jeff Green [21:47:33] Is someone around who can help me with a little problem on a private wiki and user access to it? [21:50:40] Barras: Welcome to Saturday and holiday weekends :) Which wiki is it? (just so people know if they can or can't) [21:50:54] checkuser wiki [21:51:08] ah, and the issue with it is? [21:51:39] well, private :P [21:51:58] I thought that the point of a private wiki :p [21:52:44] I need someone who can help me to get a new (old) CU to get access again to the wiki. :) [21:54:19] I don't believe ops like to get involved in these sort of matters, if there is a reason it can't be done - poking Philippe, James or Maggie might be best unless someone here is willing to deal with this. [21:54:58] you will need to lodge a request via the normal way to request access to the wiki [21:55:16] Well, I could bribe someone in the past for a same problem on a normal wiki ;) [21:55:49] Barras: bribe away but those three will likely be able to jump to a solution/ask people to do something more than a bribe ;) [21:55:51] yes but that is a normal wiki [21:56:25] It's actually only putting in a new mail address to the account [21:57:06] ah, akosiaris might be around - godog is on ops duty though. ^^ [21:59:23] Barras, they can't access the old one? or didn't set one up and don't remember the password? [22:00:02] Don't remember the password and when he left he had his old mail account closed :( [22:01:22] Yeah that's something someone with DB access can handle, presumably given approval from LCA first [22:01:23] (otherwise I wouldn't ask here for help ;-) ) [22:03:27] iirc, doing this process usually requires a prod to someone in LCA anyway so go to the core first [22:04:08] I loved the good old times when this was done the easy way :P [22:04:23] * Barras goes mailing them... [22:04:32] what was the easy way? [22:05:53] "tell me the new address. -> done" ;-) [22:06:02] some random shell user setting it by request? [22:09:50] Barras, out of interest who is this for? [22:10:44] Krenair: I won't tell you which staff member (I wouldn't have access someone random) did that in the past... and well, there aren't much new CUs being rights given today ;) [22:11:05] Reedy and MarcoAurelio? [22:11:35] Krenair: probably for the first one and yes for the second (newly appointed-old checkuser so) [22:12:17] MA [22:59:03] Just saying, my problem has been solved :) [23:00:18] Barras: who did you bribe? :o [23:00:48] top secret :P [23:01:00] :( [23:01:41] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 3 failures [23:01:41] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 48 failures [23:01:57] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: puppet fail [23:02:03] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Puppet has 34 failures [23:02:03] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: puppet fail [23:02:08] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 16 failures [23:02:09] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Puppet has 73 failures [23:02:31] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 17 failures [23:02:32] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Puppet has 60 failures [23:02:32] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet has 9 failures [23:02:32] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Puppet has 17 failures [23:02:32] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 17 failures [23:02:33] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Puppet has 15 failures [23:02:34] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Puppet has 8 failures [23:02:37] PROBLEM - puppet last run on mw1236 is CRITICAL: CRITICAL: Puppet has 38 failures [23:02:38] PROBLEM - puppetmaster backend https on strontium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [23:02:38] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 78 failures [23:02:47] PROBLEM - puppet last run on radium is CRITICAL: CRITICAL: puppet fail [23:02:49] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Puppet has 37 failures [23:02:49] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 72 failures [23:02:50] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: puppet fail [23:02:50] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 68 failures [23:02:50] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: puppet fail [23:02:50] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: puppet fail [23:02:58] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 33 failures [23:02:58] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: puppet fail [23:02:58] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: puppet fail [23:02:58] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: puppet fail [23:02:59] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: puppet fail [23:02:59] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: puppet fail [23:02:59] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: puppet fail [23:02:59] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Puppet has 20 failures [23:03:08] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: puppet fail [23:03:11] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 50 failures [23:03:11] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 11 failures [23:03:18] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Puppet has 23 failures [23:03:29] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Puppet has 19 failures [23:03:33] PROBLEM - puppet last run on rbf1001 is CRITICAL: CRITICAL: Puppet has 20 failures [23:03:33] PROBLEM - puppet last run on mw1255 is CRITICAL: CRITICAL: puppet fail [23:03:33] PROBLEM - puppet last run on elastic1020 is CRITICAL: CRITICAL: Puppet has 12 failures [23:03:34] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: puppet fail [23:03:34] PROBLEM - puppet last run on es2006 is CRITICAL: CRITICAL: puppet fail [23:03:34] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: puppet fail [23:03:34] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 9 failures [23:03:35] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 18 failures [23:03:35] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 22 failures [23:03:38] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: puppet fail [23:03:54] puppetmaster? [23:03:55] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: puppet fail [23:03:55] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 9 failures [23:03:55] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: puppet fail [23:03:55] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 16 failures [23:03:55] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: puppet fail [23:04:00] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 12 failures [23:04:08] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Puppet has 48 failures [23:04:09] PROBLEM - puppet last run on elastic1025 is CRITICAL: CRITICAL: Puppet has 19 failures [23:04:09] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 40 failures [23:04:10] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Puppet has 20 failures [23:04:10] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: puppet fail [23:04:10] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: puppet fail [23:04:10] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: puppet fail [23:04:18] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 33 failures [23:04:18] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Puppet has 22 failures [23:04:18] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: puppet fail [23:04:18] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 39 failures [23:04:18] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: puppet fail [23:04:19] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: puppet fail [23:04:29] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Puppet has 19 failures [23:04:29] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Puppet has 22 failures [23:04:30] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Puppet has 14 failures [23:04:30] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 29 failures [23:04:30] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 26 failures [23:04:30] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 25 failures [23:04:30] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: puppet fail [23:04:31] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Puppet has 21 failures [23:04:31] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: puppet fail [23:04:32] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: puppet fail [23:04:38] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: puppet fail [23:04:40] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: puppet fail [23:04:40] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: Puppet has 30 failures [23:04:40] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 24 failures [23:04:40] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: puppet fail [23:04:40] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 19 failures [23:04:41] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 14 failures [23:04:41] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Puppet has 72 failures [23:04:42] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: puppet fail [23:04:48] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: puppet fail [23:04:48] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: puppet fail [23:04:48] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 24 failures [23:05:09] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: puppet fail [23:05:10] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: puppet fail [23:05:10] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 14 failures [23:05:10] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Puppet has 19 failures [23:05:10] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: puppet fail [23:05:10] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: puppet fail [23:05:24] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: puppet fail [23:05:24] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Puppet has 65 failures [23:05:24] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 22 failures [23:05:24] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: puppet fail [23:05:24] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: puppet fail [23:05:25] PROBLEM - puppet last run on mw1241 is CRITICAL: CRITICAL: Puppet has 29 failures [23:05:25] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 72 failures [23:05:26] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 83 failures [23:05:26] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 34 failures [23:05:27] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 31 failures [23:05:38] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Puppet has 63 failures [23:05:38] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 11 failures [23:05:38] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: puppet fail [23:05:39] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Puppet has 15 failures [23:05:39] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 30 failures [23:05:39] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: puppet fail [23:05:40] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Puppet has 52 failures [23:05:40] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: puppet fail [23:05:40] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 26 failures [23:05:41] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 17 failures [23:05:41] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 28 failures [23:05:42] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Puppet has 34 failures [23:05:42] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: puppet fail [23:05:43] soo... [23:05:59] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: puppet fail [23:05:59] does ops know? [23:05:59] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: puppet fail [23:05:59] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 35 failures [23:05:59] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: puppet fail [23:05:59] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 25 failures [23:06:00] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: puppet fail [23:06:00] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Puppet has 59 failures [23:06:01] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Puppet has 20 failures [23:06:01] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Puppet has 23 failures [23:06:02] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 56 failures [23:06:02] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Puppet has 75 failures [23:06:08] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: puppet fail [23:06:28] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 29 failures [23:06:28] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: puppet fail [23:06:28] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 29 failures [23:06:28] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 76 failures [23:06:28] PROBLEM - puppet last run on radon is CRITICAL: CRITICAL: Puppet has 26 failures [23:06:29] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Puppet has 30 failures [23:06:29] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 66 failures [23:06:30] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 36 failures [23:06:30] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: puppet fail [23:06:31] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Puppet has 10 failures [23:06:31] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: puppet fail [23:06:32] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 26 failures [23:06:32] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: puppet fail [23:06:33] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 58 failures [23:06:33] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: puppet fail [23:06:34] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 77 failures [23:06:34] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: puppet fail [23:06:37] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: puppet fail [23:06:38] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 64 failures [23:06:38] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 33 failures [23:06:38] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: puppet fail [23:06:38] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 26 failures [23:06:47] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: puppet fail [23:06:47] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: puppet fail [23:06:48] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: puppet fail [23:06:48] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: puppet fail [23:06:48] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: puppet fail [23:06:48] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: puppet fail [23:07:08] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: puppet fail [23:07:08] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: puppet fail [23:07:08] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 29 failures [23:07:08] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: puppet fail [23:07:09] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Puppet has 16 failures [23:07:09] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 20 failures [23:07:19] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 18 failures [23:07:20] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [23:07:20] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Puppet has 18 failures [23:07:20] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 16 failures [23:07:20] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Puppet has 28 failures [23:07:21] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 12 failures [23:07:21] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: puppet fail [23:07:21] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: puppet fail [23:07:44] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: puppet fail [23:07:44] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 21 failures [23:07:46] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 19 failures [23:07:47] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: puppet fail [23:07:47] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Puppet has 71 failures [23:07:48] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: puppet fail [23:07:51] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: puppet fail [23:07:52] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: puppet fail [23:07:52] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: puppet fail [23:07:52] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Puppet has 23 failures [23:08:02] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 9 failures [23:08:02] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: puppet fail [23:08:02] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 60 failures [23:08:02] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: Puppet has 32 failures [23:08:02] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 23 failures [23:08:02] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet has 21 failures [23:08:02] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 13 failures [23:08:03] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: puppet fail [23:08:14] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 21 failures [23:08:15] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: puppet fail [23:08:21] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: puppet fail [23:08:25] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 31 failures [23:08:25] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: Puppet has 66 failures [23:08:25] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: puppet fail [23:08:25] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 10 failures [23:08:25] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: puppet fail [23:08:26] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: puppet fail [23:08:26] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 21 failures [23:08:27] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: puppet fail [23:08:27] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 86 failures [23:08:28] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 71 failures [23:08:28] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: Puppet has 13 failures [23:08:32] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: puppet fail [23:08:42] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: puppet fail [23:09:06] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 10 failures [23:09:06] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Puppet has 25 failures [23:09:07] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: Puppet has 33 failures [23:09:07] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: puppet fail [23:09:07] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Puppet has 38 failures [23:09:10] yeah - looks like puppetmasters (500 on strontium and palladium according to icinga) [23:09:13] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 24 failures [23:09:13] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: puppet fail [23:09:13] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 13 failures [23:09:13] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet has 38 failures [23:09:13] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: puppet fail [23:09:14] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 29 failures [23:09:14] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 21 failures [23:09:15] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: puppet fail [23:09:15] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 28 failures [23:09:16] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 12 failures [23:09:22] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: puppet fail [23:09:22] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: puppet fail [23:09:22] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: puppet fail [23:09:22] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Puppet has 34 failures [23:09:22] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Puppet has 16 failures [23:09:23] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Puppet has 44 failures [23:09:23] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: puppet fail [23:09:24] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 25 failures [23:09:30] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: puppet fail [23:09:31] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: puppet fail [23:09:32] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 19 failures [23:09:56] Not sure any opsen are around [23:09:58] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 25 failures [23:09:58] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: puppet fail [23:09:58] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Puppet has 19 failures [23:09:58] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 68 failures [23:09:58] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: puppet fail [23:09:59] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: puppet fail [23:09:59] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: puppet fail [23:10:00] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: puppet fail [23:10:00] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 10 failures [23:10:01] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 74 failures [23:10:01] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: puppet fail [23:10:03] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 17 failures [23:10:03] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 72 failures [23:10:13] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Puppet has 21 failures [23:10:13] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Puppet has 8 failures [23:10:13] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 14 failures [23:10:13] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: puppet fail [23:10:13] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Puppet has 14 failures [23:10:14] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 8 failures [23:10:14] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Puppet has 23 failures [23:10:24] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 35 failures [23:10:27] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: puppet fail [23:10:27] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: puppet fail [23:10:27] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: puppet fail [23:10:27] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: puppet fail [23:10:27] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: puppet fail [23:10:28] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: puppet fail [23:10:28] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 34 failures [23:10:29] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Puppet has 19 failures [23:10:29] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: puppet fail [23:10:30] PROBLEM - puppet last run on virt1012 is CRITICAL: CRITICAL: puppet fail [23:10:30] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 67 failures [23:10:31] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Puppet has 34 failures [23:10:31] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Puppet has 30 failures [23:10:32] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Puppet has 76 failures [23:10:32] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 13 failures [23:10:33] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 11 failures [23:10:33] PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: Puppet has 13 failures [23:10:34] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Puppet has 29 failures [23:10:34] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 12 failures [23:10:35] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: puppet fail [23:10:35] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 20 failures [23:10:36] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 27 failures [23:10:36] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 22 failures [23:10:42] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: puppet fail [23:10:54] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: puppet fail [23:10:56] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 21 failures [23:10:57] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 75 failures [23:10:57] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: puppet fail [23:10:57] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CRITICAL: Puppet has 17 failures [23:10:57] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: puppet fail [23:11:06] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 14 failures [23:11:07] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: puppet fail [23:11:17] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: puppet fail [23:11:17] PROBLEM - puppet last run on labstore2001 is CRITICAL: CRITICAL: puppet fail [23:11:17] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 12 failures [23:11:17] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: puppet fail [23:11:17] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Puppet has 26 failures [23:11:28] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: puppet fail [23:11:28] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 15 failures [23:11:28] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 18 failures [23:11:28] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: puppet fail [23:11:28] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: puppet fail [23:11:29] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: puppet fail [23:11:29] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 16 failures [23:11:30] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 26 failures [23:11:30] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 16 failures [23:11:31] PROBLEM - puppet last run on es2008 is CRITICAL: CRITICAL: Puppet has 15 failures [23:11:31] PROBLEM - puppet last run on db2039 is CRITICAL: CRITICAL: puppet fail [23:11:32] PROBLEM - puppet last run on ms-fe2004 is CRITICAL: CRITICAL: Puppet has 10 failures [23:11:32] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: puppet fail [23:11:33] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: puppet fail [23:11:33] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: Puppet has 19 failures [23:11:34] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: puppet fail [23:11:34] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: puppet fail [23:11:35] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 16 failures [23:11:35] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Puppet has 17 failures [23:11:36] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 20 failures [23:11:38] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: puppet fail [23:11:38] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet has 9 failures [23:11:48] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: puppet fail [23:11:49] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: puppet fail [23:11:49] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: puppet fail [23:11:49] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: puppet fail [23:11:58] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: puppet fail [23:11:58] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 77 failures [23:11:59] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 23 failures [23:11:59] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: puppet fail [23:11:59] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: puppet fail [23:11:59] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: puppet fail [23:11:59] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: puppet fail [23:12:00] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: puppet fail [23:12:00] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: puppet fail [23:12:01] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: puppet fail [23:12:01] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 11 failures [23:12:02] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet has 82 failures [23:12:09] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: puppet fail [23:12:09] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Puppet has 21 failures [23:12:09] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: puppet fail [23:12:17] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Puppet has 69 failures [23:12:21] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: puppet fail [23:12:21] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: puppet fail [23:12:21] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: puppet fail [23:12:21] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 12 failures [23:12:21] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: puppet fail [23:12:22] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: puppet fail [23:12:22] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 20 failures [23:12:23] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: puppet fail [23:12:23] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Puppet has 31 failures [23:12:24] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Puppet has 10 failures [23:12:24] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Puppet has 11 failures [23:12:25] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: puppet fail [23:12:26] Reedy: perfect time for you 'give-reedy-root' patch to be merged ;) [23:12:26] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 24 failures [23:12:26] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 24 failures [23:12:27] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: puppet fail [23:12:37] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: puppet fail [23:12:39] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 65 failures [23:12:39] PROBLEM - puppet last run on ms-be2013 is CRITICAL: CRITICAL: Puppet has 21 failures [23:12:39] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 15 failures [23:12:49] PROBLEM - puppet last run on es2001 is CRITICAL: CRITICAL: Puppet has 10 failures [23:12:49] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: puppet fail [23:12:58] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 30 failures [23:13:07] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 10 failures [23:13:08] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Puppet has 13 failures [23:13:08] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: puppet fail [23:13:17] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: puppet fail [23:13:19] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Puppet has 12 failures [23:13:19] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Puppet has 12 failures [23:13:19] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 80 failures [23:13:19] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: puppet fail [23:13:19] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: puppet fail [23:13:20] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: puppet fail [23:13:20] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Puppet has 34 failures [23:13:21] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 21 failures [23:13:21] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: puppet fail [23:13:22] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 34 failures [23:13:22] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: puppet fail [23:13:23] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: puppet fail [23:13:23] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 17 failures [23:13:24] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: puppet fail [23:13:27] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 80 failures [23:13:27] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 26 failures [23:13:28] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: Puppet has 17 failures [23:13:28] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: Puppet has 12 failures [23:13:28] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Puppet has 20 failures [23:13:28] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: puppet fail [23:13:28] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet has 36 failures [23:13:29] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: Puppet has 32 failures [23:13:38] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: puppet fail [23:13:39] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: puppet fail [23:13:39] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: puppet fail [23:13:45] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: puppet fail [23:13:45] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: puppet fail [23:13:46] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: puppet fail [23:13:46] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet has 11 failures [23:13:46] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 14 failures [23:13:46] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: puppet fail [23:13:46] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: puppet fail [23:13:47] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: puppet fail [23:13:47] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 28 failures [23:13:48] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: puppet fail [23:14:08] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 38 failures [23:14:08] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 17 failures [23:14:08] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: puppet fail [23:14:08] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: Puppet has 10 failures [23:14:09] PROBLEM - puppet last run on capella is CRITICAL: CRITICAL: Puppet has 11 failures [23:14:09] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: puppet fail [23:14:16] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 23 failures [23:14:16] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 27 failures [23:14:16] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: puppet fail [23:14:17] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: puppet fail [23:14:17] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 21 failures [23:14:17] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 29 failures [23:14:17] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 26 failures [23:14:18] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 72 failures [23:14:18] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: puppet fail [23:14:19] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: puppet fail [23:14:19] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 68 failures [23:14:20] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: puppet fail [23:14:20] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: puppet fail [23:14:21] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 67 failures [23:14:21] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: puppet fail [23:14:22] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Puppet has 9 failures [23:14:22] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: puppet fail [23:14:23] PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: Puppet has 25 failures [23:14:28] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: puppet fail [23:14:28] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: puppet fail [23:14:28] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 34 failures [23:14:28] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: puppet fail [23:14:28] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 72 failures [23:14:29] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: puppet fail [23:14:29] PROBLEM - puppet last run on db2009 is CRITICAL: CRITICAL: Puppet has 14 failures [23:14:30] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Puppet has 20 failures [23:14:30] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: puppet fail [23:14:31] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: puppet fail [23:14:31] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 26 failures [23:14:32] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: puppet fail [23:14:32] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: puppet fail [23:14:33] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: puppet fail [23:14:47] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: puppet fail [23:14:47] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: puppet fail [23:14:47] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 9 failures [23:14:47] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: Puppet has 21 failures [23:15:01] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: puppet fail [23:15:01] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: puppet fail [23:15:02] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: puppet fail [23:15:02] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 13 failures [23:15:14] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [23:15:14] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: puppet fail [23:15:14] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 54 failures [23:15:18] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 13 failures [23:15:18] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: puppet fail [23:15:18] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 34 failures [23:15:18] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 77 failures [23:15:18] PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [23:15:19] PROBLEM - puppet last run on db2002 is CRITICAL: CRITICAL: Puppet has 12 failures [23:15:19] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: puppet fail [23:15:20] PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: puppet fail [23:15:20] PROBLEM - puppet last run on ms-be2004 is CRITICAL: CRITICAL: puppet fail [23:15:21] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet has 14 failures [23:15:21] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: puppet fail [23:15:27] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: puppet fail [23:15:39] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 25 failures [23:15:39] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 19 failures [23:15:39] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: puppet fail [23:15:39] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: puppet fail [23:15:39] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: puppet fail [23:15:40] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Puppet has 25 failures [23:15:40] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 14 failures [23:15:41] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 22 failures [23:15:41] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: puppet fail [23:15:42] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 18 failures [23:15:51] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: puppet fail [23:15:59] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 15 failures [23:15:59] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: puppet fail [23:15:59] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: puppet fail [23:16:00] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 68 failures [23:16:00] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 21 failures [23:16:00] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: puppet fail [23:16:13] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 11 failures [23:16:13] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: puppet fail [23:16:13] PROBLEM - puppet last run on mw1228 is CRITICAL: CRITICAL: Puppet has 32 failures [23:16:13] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 70 failures [23:16:13] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: puppet fail [23:16:14] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: puppet fail [23:16:14] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: puppet fail [23:16:15] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: puppet fail [23:16:15] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: puppet fail [23:16:16] PROBLEM - puppet last run on mw1235 is CRITICAL: CRITICAL: Puppet has 78 failures [23:16:16] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: puppet fail [23:16:17] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: puppet fail [23:16:19] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 59 failures [23:16:20] PROBLEM - puppet last run on elastic1022 is CRITICAL: CRITICAL: puppet fail [23:16:20] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: puppet fail [23:16:20] PROBLEM - puppet last run on mw1222 is CRITICAL: CRITICAL: Puppet has 67 failures [23:16:20] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: puppet fail [23:16:31] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: puppet fail [23:16:32] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 11 failures [23:16:36] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 22 failures [23:16:36] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 13 failures [23:16:36] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 34 failures [23:16:37] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 27 failures [23:16:37] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 11 failures [23:16:37] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: puppet fail [23:16:37] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Puppet has 74 failures [23:16:38] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: puppet fail [23:16:38] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: puppet fail [23:16:39] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: puppet fail [23:16:39] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 21 failures [23:16:40] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Puppet has 19 failures [23:16:40] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 18 failures [23:16:41] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: puppet fail [23:16:41] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 13 failures [23:16:42] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 34 failures [23:16:42] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: puppet fail [23:16:57] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: puppet fail [23:16:57] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: puppet fail [23:17:03] PROBLEM - puppet last run on db2029 is CRITICAL: CRITICAL: Puppet has 21 failures [23:17:03] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: puppet fail [23:17:03] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 10 failures [23:17:03] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: puppet fail [23:17:03] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: puppet fail [23:17:04] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Puppet has 17 failures [23:17:04] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 11 failures [23:17:21] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: Puppet has 14 failures [23:17:21] PROBLEM - puppet last run on elastic1019 is CRITICAL: CRITICAL: puppet fail [23:17:21] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Puppet has 12 failures [23:17:21] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 24 failures [23:17:21] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Puppet has 24 failures [23:17:31] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: puppet fail [23:17:31] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: puppet fail [23:17:31] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: puppet fail [23:17:31] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet has 52 failures [23:17:31] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: puppet fail [23:17:32] PROBLEM - puppet last run on lithium is CRITICAL: CRITICAL: Puppet has 17 failures [23:17:32] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: puppet fail [23:17:33] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: puppet fail [23:17:33] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: puppet fail [23:17:34] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: puppet fail [23:17:34] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Puppet has 20 failures [23:17:35] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: Puppet has 8 failures [23:17:37] Really not sure it's worth paging anyone... [23:17:40] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: puppet fail [23:17:41] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Puppet has 15 failures [23:17:41] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: puppet fail [23:17:41] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 20 failures [23:17:41] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: puppet fail [23:17:41] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 31 failures [23:17:41] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 35 failures [23:17:42] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 21 failures [23:17:42] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 36 failures [23:17:43] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Puppet has 8 failures [23:17:43] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Puppet has 30 failures [23:17:44] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: puppet fail [23:17:44] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: puppet fail [23:17:45] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: puppet fail [23:17:45] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: Puppet has 22 failures [23:17:46] PROBLEM - puppet last run on virt1010 is CRITICAL: CRITICAL: puppet fail [23:17:46] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: puppet fail [23:17:47] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: puppet fail [23:17:47] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: puppet fail [23:17:48] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: Puppet has 20 failures [23:17:48] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 64 failures [23:17:49] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Puppet has 64 failures [23:17:49] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 20 failures [23:17:50] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 49 failures [23:17:50] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 16 failures [23:17:51] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: puppet fail [23:17:51] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: puppet fail [23:18:04] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet has 38 failures [23:18:04] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Puppet has 8 failures [23:18:04] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 24 failures [23:18:04] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 11 failures [23:18:04] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: puppet fail [23:18:04] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Puppet has 14 failures [23:18:05] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Puppet has 20 failures [23:18:05] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 26 failures [23:18:13] Ahh!!! The puppets are failing everywhere! [23:18:16] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 16 failures [23:18:16] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 14 failures [23:18:16] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: puppet fail [23:18:16] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: puppet fail [23:18:16] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Puppet has 78 failures [23:18:16] PROBLEM - puppet last run on db2001 is CRITICAL: CRITICAL: Puppet has 10 failures [23:18:17] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Puppet has 23 failures [23:18:17] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Puppet has 12 failures [23:18:18] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: puppet fail [23:18:18] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: puppet fail [23:18:25] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: puppet fail [23:18:26] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: Puppet has 24 failures [23:18:26] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Puppet has 20 failures [23:18:26] PROBLEM - puppet last run on mw1249 is CRITICAL: CRITICAL: Puppet has 68 failures [23:18:26] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: puppet fail [23:18:26] PROBLEM - puppet last run on plutonium is CRITICAL: CRITICAL: Puppet has 29 failures [23:18:27] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: puppet fail [23:18:27] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 77 failures [23:18:28] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 15 failures [23:18:28] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 62 failures [23:18:29] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: puppet fail [23:18:29] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Puppet has 47 failures [23:18:30] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 12 failures [23:18:30] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Puppet has 66 failures [23:18:31] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: puppet fail [23:18:35] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: puppet fail [23:18:35] PROBLEM - puppet last run on db2004 is CRITICAL: CRITICAL: Puppet has 22 failures [23:18:35] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Puppet has 12 failures [23:18:35] PROBLEM - puppet last run on elastic1015 is CRITICAL: CRITICAL: puppet fail [23:18:35] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet has 23 failures [23:18:36] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: puppet fail [23:18:36] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Puppet has 11 failures [23:18:37] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Puppet has 7 failures [23:18:37] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Puppet has 54 failures [23:18:44] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: puppet fail [23:18:47] PROBLEM - puppet last run on mw1247 is CRITICAL: CRITICAL: puppet fail [23:18:47] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: puppet fail [23:18:48] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: puppet fail [23:18:48] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: puppet fail [23:18:48] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: puppet fail [23:18:48] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Puppet has 13 failures [23:18:48] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Puppet has 15 failures [23:18:49] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 31 failures [23:18:49] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 23 failures [23:18:50] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: puppet fail [23:18:56] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: puppet fail [23:18:57] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: puppet fail [23:18:57] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: puppet fail [23:18:57] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: puppet fail [23:18:57] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: puppet fail [23:18:57] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: puppet fail [23:18:57] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 8 failures [23:18:58] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Puppet has 11 failures [23:19:00] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: puppet fail [23:19:09] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Puppet has 19 failures [23:19:10] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Puppet has 10 failures [23:19:10] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: puppet fail [23:19:10] PROBLEM - puppet last run on db2037 is CRITICAL: CRITICAL: puppet fail [23:19:17] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: puppet fail [23:19:19] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 29 failures [23:19:20] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.058 second response time [23:19:20] PROBLEM - puppet last run on db2016 is CRITICAL: CRITICAL: Puppet has 22 failures [23:19:24] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: puppet fail [23:19:24] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: puppet fail [23:19:24] PROBLEM - puppet last run on lvs2006 is CRITICAL: CRITICAL: puppet fail [23:19:24] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: puppet fail [23:19:24] PROBLEM - puppet last run on haedus is CRITICAL: CRITICAL: Puppet has 10 failures [23:19:25] PROBLEM - puppet last run on ms-be2001 is CRITICAL: CRITICAL: puppet fail [23:19:25] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: puppet fail [23:19:26] PROBLEM - puppet last run on ms-be2008 is CRITICAL: CRITICAL: puppet fail [23:19:26] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 25 failures [23:19:27] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Puppet has 32 failures [23:19:36] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Puppet has 24 failures [23:19:37] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Puppet has 18 failures [23:19:45] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: puppet fail [23:19:46] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: puppet fail [23:19:46] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: puppet fail [23:19:46] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: puppet fail [23:20:14] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 24 failures [23:20:14] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet has 10 failures [23:20:15] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: puppet fail [23:20:15] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 25 failures [23:20:19] PROBLEM - puppet last run on elastic1023 is CRITICAL: CRITICAL: puppet fail [23:20:22] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: puppet fail [23:20:23] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: puppet fail [23:20:23] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet has 40 failures [23:20:23] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: puppet fail [23:20:23] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: puppet fail [23:20:23] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Puppet has 17 failures [23:20:23] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 20 failures [23:20:24] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: puppet fail [23:20:24] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: puppet fail [23:20:25] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 26 failures [23:20:25] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: puppet fail [23:20:26] PROBLEM - puppet last run on mw1225 is CRITICAL: CRITICAL: puppet fail [23:20:26] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: Puppet has 13 failures [23:20:27] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: puppet fail [23:20:27] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Puppet has 24 failures [23:20:28] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: puppet fail [23:20:28] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Puppet has 14 failures [23:20:32] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: puppet fail [23:20:53] PROBLEM - puppet last run on lvs2003 is CRITICAL: CRITICAL: puppet fail [23:20:53] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: puppet fail [23:20:53] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: puppet fail [23:20:53] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Puppet has 23 failures [23:20:53] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 10 failures [23:20:53] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: puppet fail [23:20:53] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 22 failures [23:20:54] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: Puppet has 7 failures [23:20:54] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Puppet has 61 failures [23:20:55] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 19 failures [23:20:55] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: puppet fail [23:20:56] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: puppet fail [23:20:56] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: puppet fail [23:20:57] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 64 failures [23:20:57] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Puppet has 36 failures [23:20:58] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: puppet fail [23:20:58] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: puppet fail [23:20:59] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: puppet fail [23:21:04] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Puppet has 24 failures [23:21:05] PROBLEM - puppet last run on ms-be2012 is CRITICAL: CRITICAL: puppet fail [23:21:05] PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: puppet fail [23:21:05] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 19 failures [23:21:14] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: puppet fail [23:21:16] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Puppet has 20 failures [23:21:16] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 13 failures [23:21:31] Reedy: yeah, not worth a page really [23:21:33] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: puppet fail [23:21:33] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 60 failures [23:21:33] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 25 failures [23:21:33] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Puppet has 24 failures [23:21:33] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet has 30 failures [23:21:34] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: puppet fail [23:21:34] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet has 36 failures [23:21:35] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 27 failures [23:21:43] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Puppet has 30 failures [23:21:44] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Puppet has 27 failures [23:21:44] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 36 failures [23:21:44] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Puppet has 69 failures [23:21:44] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 36 failures [23:21:44] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: Puppet has 52 failures [23:21:45] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: puppet fail [23:21:45] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: puppet fail [23:21:45] puppetmasters should recover eventually as they have in the past somehow [23:21:46] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: puppet fail [23:21:46] PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: Puppet has 72 failures [23:21:47] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 34 failures [23:21:47] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: Puppet has 69 failures [23:21:48] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: puppet fail [23:22:03] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: puppet fail [23:22:03] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: puppet fail [23:22:03] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Puppet has 29 failures [23:22:03] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: puppet fail [23:22:03] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: puppet fail [23:22:04] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: puppet fail [23:22:04] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: puppet fail [23:22:05] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: puppet fail [23:22:05] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Puppet has 10 failures [23:22:06] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: puppet fail [23:22:06] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: puppet fail [23:22:15] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: puppet fail [23:22:15] PROBLEM - puppet last run on db2003 is CRITICAL: CRITICAL: puppet fail [23:22:15] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: puppet fail [23:22:15] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 17 failures [23:22:35] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: puppet fail [23:22:36] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Puppet has 67 failures [23:22:36] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 16 failures [23:22:36] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Puppet has 28 failures [23:22:36] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: puppet fail [23:22:36] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: puppet fail [23:22:37] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Puppet has 16 failures [23:22:37] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: puppet fail [23:22:38] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: puppet fail [23:22:38] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: Puppet has 66 failures [23:22:39] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: puppet fail [23:22:39] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: puppet fail [23:22:40] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Puppet has 69 failures [23:22:40] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Puppet has 9 failures [23:22:41] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: puppet fail [23:22:41] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 22 failures [23:22:43] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: puppet fail [23:22:43] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: puppet fail [23:22:43] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: puppet fail [23:22:43] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: puppet fail [23:22:44] PROBLEM - puppet last run on search1013 is CRITICAL: CRITICAL: puppet fail [23:22:46] PROBLEM - puppet last run on mw1223 is CRITICAL: CRITICAL: puppet fail [23:22:47] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: puppet fail [23:22:54] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: puppet fail [23:22:54] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Puppet has 25 failures [23:23:04] PROBLEM - puppet last run on ms-fe2002 is CRITICAL: CRITICAL: puppet fail [23:23:13] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: Puppet has 37 failures [23:23:13] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Puppet has 20 failures [23:23:13] PROBLEM - puppet last run on einsteinium is CRITICAL: CRITICAL: puppet fail [23:23:13] PROBLEM - puppet last run on iridium is CRITICAL: CRITICAL: Puppet has 26 failures [23:23:14] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: puppet fail [23:23:14] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Puppet has 12 failures [23:23:14] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: puppet fail [23:23:15] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 21 failures [23:23:15] PROBLEM - puppet last run on db2011 is CRITICAL: CRITICAL: Puppet has 16 failures [23:23:16] PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: Puppet has 9 failures [23:23:24] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 15 failures [23:23:25] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: puppet fail [23:23:43] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 18 failures [23:23:48] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: puppet fail [23:23:52] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 67 failures [23:23:52] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: puppet fail [23:23:53] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: puppet fail [23:23:53] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: puppet fail [23:23:53] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 41 failures [23:23:54] PROBLEM - puppet last run on mw1231 is CRITICAL: CRITICAL: Puppet has 26 failures [23:23:54] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet has 18 failures [23:23:54] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 52 failures [23:24:15] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 56 failures [23:24:15] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 69 failures [23:24:15] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 74 failures [23:24:25] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Puppet has 9 failures [23:24:34] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Puppet has 39 failures [23:24:44] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: puppet fail [23:25:14] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 69 failures [23:25:14] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Puppet has 11 failures [23:26:07] PROBLEM - puppet last run on virt1011 is CRITICAL: CRITICAL: Puppet has 10 failures [23:26:14] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Puppet has 10 failures [23:28:06] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:32:04] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: puppet fail [23:33:55] !log restarting puppetmasters [23:34:01] Logged the message, Master [23:35:45] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:35:49] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:36:03] RECOVERY - puppetmaster backend https on strontium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.022 second response time [23:36:14] RECOVERY - puppet last run on radium is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:36:25] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:36:27] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:36:27] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:36:27] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:36:36] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:36:36] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:36:46] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:36:49] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:36:49] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:37:04] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:05] RECOVERY - puppet last run on mw1255 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:37:05] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:37:05] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:37:05] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:05] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:37:05] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:37:16] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:37:25] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:37:25] RECOVERY - puppet last run on elastic1025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:38] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:38] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:37:38] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:44] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:45] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:45] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:45] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:45] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:45] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:37:46] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:54] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:37:55] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:37:55] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:55] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:37:55] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:37:55] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:37:55] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:37:56] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:37:56] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:24] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:29] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:38:29] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:29] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:38:29] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:38:36] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:38:36] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:36] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:36] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:38:36] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:37] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:50] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:50] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:51] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:38:51] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:38:51] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:38:51] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:51] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:52] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:38:52] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:53] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:38:53] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:54] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:38:54] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:38:55] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:07] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:07] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:07] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:13] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:13] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:39:13] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:39:13] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:39:13] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:13] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:14] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:15] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:15] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:26] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:26] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:26] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:26] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:26] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:27] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:39:27] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:28] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:28] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:39:29] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:29] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:30] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:30] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:31] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:31] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:32] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:32] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:35] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:37] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:37] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:38] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:54] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:54] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:55] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:55] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:55] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:39:55] RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:55] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:56] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:56] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:57] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:39:57] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:58] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:39:58] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:39:59] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:06] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:06] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:06] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:06] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:06] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:40:07] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:40:07] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:40:08] RECOVERY - puppet last run on virt1012 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:40:08] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:09] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:40:09] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:40:10] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:40:10] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:11] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:11] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:40:15] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:19] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:26] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:40:26] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:40:26] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:26] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:40:26] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:40:27] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:40:35] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:40] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:40:40] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:40:41] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:40:45] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:45] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:40:55] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:40:55] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:55] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:40:56] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:40:56] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:40:56] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:40:56] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:05] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:06] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:06] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:41:06] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:06] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:06] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:07] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:15] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:16] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:16] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:16] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:41:16] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:16] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:17] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:17] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:18] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:18] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:41:24] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:24] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:24] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:24] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:24] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:25] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:35] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:35] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:36] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:36] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:41:36] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:36] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:37] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:41:37] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:41:37] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:41:38] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:38] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:41:44] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:44] RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:41:44] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:45] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:45] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:41:45] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:45] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:45] paravoid, any idea why puppetmaster randomly dies like that? [23:41:46] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:46] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:47] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:41:53] Krenair: nope [23:41:59] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:41:59] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:01] Krenair: and I'm not going to debug it now [23:42:05] :) [23:42:09] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:42:09] :) [23:42:10] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:42:10] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:10] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:42:10] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:42:15] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:16] RECOVERY - puppet last run on ms-be2013 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:42:16] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:16] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:16] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:21] just that restarting it "fixes" the issue. strange [23:42:27] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:42:27] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:38] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:42:47] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:47] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:42:47] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:42:47] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:42:47] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:42:57] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:57] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:42:57] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:42:58] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:42:58] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:42:58] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:42:58] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:42:59] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:42:59] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:00] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:00] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:43:01] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:01] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:02] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:02] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:03] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:03] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:04] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:43:04] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:17] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:17] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:17] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:43:17] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:17] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:18] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:43:18] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:19] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:43:19] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:20] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:43:20] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:21] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:37] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:37] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:37] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:43:37] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [23:43:38] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:43:38] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:38] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:39] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:39] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:40] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:43:40] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:43:41] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:41] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:42] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:42] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:43] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:45] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:46] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:46] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:46] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:43:46] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:46] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:46] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:43:50] RECOVERY - puppet last run on db2009 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:43:57] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:43:57] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:43:58] RECOVERY - puppet last run on mw1224 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:43:58] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:58] RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:44:01] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:44:01] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:44:01] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:44:10] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:10] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:44:10] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:44:10] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:44:10] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:44:11] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:44:11] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:17] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:44:18] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:19] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:44:28] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:44:28] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:38] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:38] RECOVERY - puppet last run on db2002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:44:38] RECOVERY - puppet last run on labstore2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:38] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:44:38] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:44:38] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:39] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:39] RECOVERY - puppet last run on es2008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:40] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:40] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:44:41] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:44:41] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:42] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:44:48] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:44:54] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:54] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:44:54] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:54] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:54] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:55] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:55] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:44:56] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:45:07] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:08] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:08] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:45:08] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:45:08] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:45:20] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:45:22] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:45:22] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:22] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [23:45:22] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:45:22] RECOVERY - puppet last run on mw1228 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:45:23] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:23] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:45:24] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:24] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:25] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:45:25] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:45:26] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:26] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:27] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:45:27] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:45:28] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:36] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:36] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:43] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:45:43] RECOVERY - puppet last run on mw1222 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:45:43] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:43] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:45:43] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:44] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:45:44] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:45] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:45] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:46] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:46] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:45:47] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:45:47] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:45:48] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:48] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:49] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:49] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:50] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:50] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:51] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:45:51] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:45:56] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:02] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:08] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:46:27] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:46:27] RECOVERY - puppet last run on es2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:46:27] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:27] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:46:27] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:38] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:38] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:38] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:38] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:46:38] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:46] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:52] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:46:52] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:46:52] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:52] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:52] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:46:53] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:46:53] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:54] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:46:58] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:58] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:59] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:59] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:59] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:46:59] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:46:59] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:00] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:06] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:11] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:47:11] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:11] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:11] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:11] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:47:12] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:12] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:13] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:13] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:14] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:14] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:47:15] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:17] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:17] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:17] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:26] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:26] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:47:26] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:27] RECOVERY - puppet last run on capella is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:27] RECOVERY - puppet last run on db2005 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:27] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:39] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:39] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:46] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:48] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:47:48] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:48] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:48] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:48] RECOVERY - puppet last run on mw1249 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:47:49] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:49] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:50] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:50] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:51] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:51] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:52] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:47:52] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:53] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:53] RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:56] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:56] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:56] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:56] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:57] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:57] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:57] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:47:58] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:58] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:47:59] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:47:59] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:00] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:00] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:01] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:10] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:10] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:10] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:48:21] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:21] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:48:21] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:30] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:30] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:48:30] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:30] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:31] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:39] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:40] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:41] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:41] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:48:41] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:48:41] RECOVERY - puppet last run on db2016 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:48:41] RECOVERY - puppet last run on ms-be2004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:41] RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:42] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:42] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:43] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:43] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:49] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:51] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:48:51] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:51] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:48:51] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:48:51] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:52] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:48:52] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:53] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:48:53] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:01] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:01] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:01] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:01] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:49:11] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:49:12] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:12] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:49:12] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:12] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:12] RECOVERY - puppet last run on mw1235 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:12] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:13] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:49:13] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:14] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:14] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:49:15] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:15] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:16] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:16] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:49:21] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:49:21] RECOVERY - puppet last run on elastic1022 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:21] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:21] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:21] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:30] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:30] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:30] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:30] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:49:30] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:49:31] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:31] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:39] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:39] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:39] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:49:40] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:49:40] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:49:40] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:40] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:41] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:49:50] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:49:52] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:52] RECOVERY - puppet last run on db2029 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:49:52] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:50:00] RECOVERY - puppet last run on ms-be2012 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [23:50:00] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:50:00] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:00] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:00] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:50:11] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:11] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:50:30] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:30] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:30] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:31] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:31] RECOVERY - puppet last run on lithium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:31] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:31] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:32] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:37] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:37] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:37] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:40] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:41] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:41] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:41] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:50:41] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:41] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:42] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:42] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:51] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:51] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:50:51] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:51] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:51] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:50:52] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:52] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:53] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:50:53] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:54] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:54] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:55] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:50:55] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:50:56] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:56] RECOVERY - puppet last run on virt1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:57] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:57] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:50:58] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:58] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:50:59] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:05] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:05] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:51:05] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:51:05] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:05] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:51:06] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:51:06] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [23:51:07] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:51:07] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:08] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:08] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:51:09] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:51:09] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [23:51:12] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:51:12] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:24] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:24] RECOVERY - puppet last run on db2003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:51:24] RECOVERY - puppet last run on db2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:24] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:25] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:25] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:51:25] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:51:30] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:30] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:30] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:30] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:30] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:51:31] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:51:31] RECOVERY - puppet last run on plutonium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:35] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:35] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:35] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:51:35] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:51:35] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:36] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:36] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:51:37] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:51:42] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:42] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:50] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:51:56] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:56] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:51:58] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:59] RECOVERY - puppet last run on db2004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:51:59] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:00] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:52:00] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:52:00] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:00] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:00] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:02] RECOVERY - puppet last run on mw1247 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:02] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:02] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:02] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:03] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:03] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:11] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:15] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:18] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:19] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:19] RECOVERY - puppet last run on acamar is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:19] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:25] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:25] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:52:25] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:25] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:25] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:26] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:26] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:52:27] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:36] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:36] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:43] RECOVERY - puppet last run on db2037 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:43] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:46] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:47] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:52:47] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [23:52:47] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:55] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:58] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:58] RECOVERY - puppet last run on lvs2006 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:52:58] RECOVERY - puppet last run on db2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:52:58] RECOVERY - puppet last run on ms-be2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:52:59] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:59] RECOVERY - puppet last run on haedus is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:52:59] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:00] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:00] RECOVERY - puppet last run on ms-be2008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:01] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:01] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:53:02] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:06] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:06] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:06] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:06] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:06] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:15] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:53:15] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:15] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:15] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:15] RECOVERY - puppet last run on elastic1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:16] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:16] RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:27] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:27] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:28] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:28] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:53:36] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:36] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:36] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:36] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:36] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:37] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:37] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [23:53:38] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:38] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:39] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:39] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:40] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:53:40] RECOVERY - puppet last run on lvs2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:41] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:53:41] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:42] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:42] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:56] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:53:56] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:56] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:57] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:57] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:57] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:57] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:53:58] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:58] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:59] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:53:59] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:00] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:00] RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:54:01] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:54:18] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:26] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:54:35] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:54:36] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:54:36] RECOVERY - puppet last run on mw1241 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:54:46] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:54:49] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [23:54:49] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:54:49] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:49] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:49] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:50] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:54:50] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:54:51] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:54:51] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:52] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:54:52] RECOVERY - puppet last run on mw1229 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:54:53] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:54:56] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:55:03] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:55:09] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:55:19] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:55:22] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:55:22] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:55:22] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:22] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:55:22] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:55:23] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:23] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:55:24] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:24] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:55:25] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:55:30] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:55:33] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:33] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:55:34] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:55:34] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:55:34] RECOVERY - puppet last run on mw1236 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:40] RECOVERY - puppet last run on virt1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:40] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:55:40] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:55:54] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:03] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:05] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:05] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:56:05] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:05] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:05] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:56:06] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:06] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:07] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:07] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:56:08] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:08] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:56:09] RECOVERY - puppet last run on mw1223 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:09] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:19] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:30] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:33] RECOVERY - puppet last run on ms-fe2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:33] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:43] RECOVERY - puppet last run on einsteinium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:44] RECOVERY - puppet last run on iridium is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:44] RECOVERY - puppet last run on elastic1020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:44] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:44] RECOVERY - puppet last run on baham is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:56:50] RECOVERY - puppet last run on db2028 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:50] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:56:50] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:59] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:56:59] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:57:22] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:57:22] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:22] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:23] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:23] RECOVERY - puppet last run on mw1231 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:23] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:23] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:23] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:24] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:57:39] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:40] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:57:51] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:58:39] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:58:49] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures