[00:00:04] PROBLEM - SSH on fenari is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:01:03] RECOVERY - SSH on fenari is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [00:01:32] PROBLEM - DPKG on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:02:25] RECOVERY - DPKG on fenari is OK: All packages OK [00:02:28] Certainly looks like it. [00:02:56] apache was going nuts there earlier [00:03:03] PROBLEM - check if dhclient is running on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:03:03] PROBLEM - nutcracker process on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:03:03] PROBLEM - nutcracker port on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:03:03] PROBLEM - Disk space on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:03:03] PROBLEM - check configured eth on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:03:32] https://ganglia.wikimedia.org/latest/?r=2hr&cs=&ce=&c=Miscellaneous+pmtpa&h=fenari.wikimedia.org&tab=m&vn=&hide-hf=false&metric_group= :/ [00:04:03] RECOVERY - check configured eth on fenari is OK: NRPE: Unable to read output [00:04:03] RECOVERY - check if dhclient is running on fenari is OK: PROCS OK: 0 processes with command name dhclient [00:04:03] RECOVERY - nutcracker process on fenari is OK: PROCS OK: 1 process with UID = 116 (nutcracker), command name nutcracker [00:04:03] RECOVERY - Disk space on fenari is OK: DISK OK [00:04:03] RECOVERY - nutcracker port on fenari is OK: TCP OK - 0.000 second response time on port 11212 [00:04:13] Looks not goodly. [00:05:35] PROBLEM - DPKG on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:07:23] RECOVERY - DPKG on fenari is OK: All packages OK [00:08:12] PROBLEM - check configured eth on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:10:16] PROBLEM - nutcracker port on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:10:22] PROBLEM - check if dhclient is running on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:10:22] PROBLEM - nutcracker process on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:10:22] PROBLEM - Disk space on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:10:33] PROBLEM - DPKG on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:11:00] bd808, what does fenari actually serve anyway? [00:11:09] noc.wm.o [00:11:12] RECOVERY - nutcracker port on fenari is OK: TCP OK - 0.000 second response time on port 11212 [00:11:12] RECOVERY - Disk space on fenari is OK: DISK OK [00:11:12] RECOVERY - check configured eth on fenari is OK: NRPE: Unable to read output [00:11:26] and it's also a login bastion [00:11:46] ah. I had assumed noc was moved to eqiad [00:11:55] not yet :( [00:12:04] RECOVERY - nutcracker process on fenari is OK: PROCS OK: 1 process with UID = 116 (nutcracker), command name nutcracker [00:12:04] RECOVERY - check if dhclient is running on fenari is OK: PROCS OK: 0 processes with command name dhclient [00:12:12] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:12:13] I think they are actively working on that now though [00:12:23] s/they/ops/ [00:12:38] user homes are moving to terbium with people.wm.o now [00:13:48] yeah but there's still other stuff on noc [00:14:35] RECOVERY - DPKG on fenari is OK: All packages OK [00:15:13] RECOVERY - RAID on fenari is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [00:18:32] PROBLEM - RAID on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:18:32] PROBLEM - check configured eth on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:19:42] PROBLEM - DPKG on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:20:32] PROBLEM - nutcracker port on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:20:33] PROBLEM - check if dhclient is running on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:21:42] RECOVERY - DPKG on fenari is OK: All packages OK [00:23:32] RECOVERY - check if dhclient is running on fenari is OK: PROCS OK: 0 processes with command name dhclient [00:23:32] RECOVERY - nutcracker port on fenari is OK: TCP OK - 0.000 second response time on port 11212 [00:27:32] RECOVERY - check configured eth on fenari is OK: NRPE: Unable to read output [00:29:31] !log restarting apache2 on fenari [00:29:32] RECOVERY - RAID on fenari is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [00:29:32] RECOVERY - HTTP on fenari is OK: HTTP OK: HTTP/1.1 200 OK - 4775 bytes in 0.078 second response time [00:29:38] Logged the message, Master [00:31:45] RECOVERY - Disk space on elastic1009 is OK: DISK OK [00:36:42] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [00:45:52] !log fenari appears to still have twemproxy (in addition to nutcracker); decom'ing. [00:45:57] Logged the message, Master [00:53:23] someone is scraping the site [00:53:43] noc? [00:54:01] yeah [00:54:05] lol, why? [00:54:18] there's what... user homes and stuff in the public repo? [00:58:35] they stopped [00:58:40] i wonder if they're here :P [01:59:53] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Epic puppet fail [02:01:28] !log LocalisationUpdate failed: mwversionsinuse returned empty list [02:01:39] Logged the message, Master [02:09:28] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3610 MB (3% inode=99%): [02:15:05] nl-wiki is completely down [02:15:31] 403 Forbidden everywhere [02:15:33] 403 [02:15:51] whose on duty? [02:16:04] apergos: ? [02:16:16] Forbidden [02:16:17] You don't have permission to access /w/index.php on this server. [02:16:27] enWS, and other reports coming in [02:16:28] thanks for fixing it (?) [02:16:29] PROBLEM - Apache HTTP on mw1075 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1415 bytes in 0.019 second response time [02:16:39] PROBLEM - Apache HTTP on mw1100 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1415 bytes in 0.012 second response time [02:16:42] oh no [02:16:49] PROBLEM - Apache HTTP on mw1073 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1415 bytes in 0.019 second response time [02:16:51] seems to be working again now [02:16:58] but icinga-wm is lighting up [02:17:10] PROBLEM - Apache HTTP on mw1040 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 1415 bytes in 0.020 second response time [02:17:23] Might be delayed a wee bit :) [02:17:30] RECOVERY - Apache HTTP on mw1075 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.080 second response time [02:17:31] ganglia shows a bunch of important graphs dropped [02:17:39] RECOVERY - Apache HTTP on mw1100 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.108 second response time [02:17:51] Krenair: what caused all the 403s? [02:17:52] RECOVERY - Apache HTTP on mw1073 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.055 second response time [02:18:00] users are quicker than icinga [02:18:10] RECOVERY - Apache HTTP on mw1040 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.090 second response time [02:18:24] eqiad app servers hit 0 bytes/sec network :S [02:19:29] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:00:10] RECOVERY - Disk space on virt0 is OK: DISK OK [04:43:56] (03PS1) 10Ori.livneh: nginx: drop 'enabled' parameter [puppet/nginx] - 10https://gerrit.wikimedia.org/r/160256 [04:43:58] (03CR) 10jenkins-bot: [V: 04-1] nginx: drop 'enabled' parameter [puppet/nginx] - 10https://gerrit.wikimedia.org/r/160256 (owner: 10Ori.livneh) [04:45:35] (03PS2) 10Ori.livneh: nginx: drop 'enabled' parameter [puppet/nginx] - 10https://gerrit.wikimedia.org/r/160256 [04:45:37] (03CR) 10jenkins-bot: [V: 04-1] nginx: drop 'enabled' parameter [puppet/nginx] - 10https://gerrit.wikimedia.org/r/160256 (owner: 10Ori.livneh) [04:46:10] (03PS3) 10Ori.livneh: nginx: drop 'enabled' parameter [puppet/nginx] - 10https://gerrit.wikimedia.org/r/160256 [05:46:09] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Epic puppet fail [05:47:10] PROBLEM - Host analytics1028 is DOWN: PING CRITICAL - Packet loss = 100% [05:49:21] RECOVERY - Host analytics1028 is UP: PING OK - Packet loss = 0%, RTA = 1.18 ms [06:06:23] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:27:41] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Epic puppet fail [06:30:21] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:21] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:11] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Epic puppet fail [06:33:31] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Epic puppet fail [06:33:42] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Epic puppet fail [06:33:42] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Epic puppet fail [06:34:03] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Epic puppet fail [06:34:04] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Epic puppet fail [06:34:11] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Epic puppet fail [06:34:13] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Epic puppet fail [06:39:12] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Epic puppet fail [06:39:12] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Epic puppet fail [06:39:12] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 2.581 second response time [06:39:24] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Epic puppet fail [06:39:24] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Epic puppet fail [06:39:24] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Epic puppet fail [06:39:41] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Epic puppet fail [06:39:43] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Epic puppet fail [06:40:11] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Epic puppet fail [06:40:21] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Epic puppet fail [06:40:22] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Epic puppet fail [06:40:31] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Epic puppet fail [06:40:41] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Epic puppet fail [06:40:42] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Epic puppet fail [06:40:42] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Epic puppet fail [06:42:31] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:42:31] RECOVERY - Disk space on ms1004 is OK: DISK OK [06:44:01] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Epic puppet fail [06:44:21] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Epic puppet fail [06:44:22] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Epic puppet fail [06:45:04] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Epic puppet fail [06:45:11] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Epic puppet fail [06:45:11] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Epic puppet fail [06:45:27] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Epic puppet fail [06:45:31] PROBLEM - Disk space on ms1004 is CRITICAL: DISK CRITICAL - free space: / 1 MB (0% inode=94%): /var/lib/ureadahead/debugfs 1 MB (0% inode=94%): [06:45:41] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:42] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Epic puppet fail [06:45:53] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: Epic puppet fail [06:46:01] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Epic puppet fail [06:46:31] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:31] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Epic puppet fail [06:46:42] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Epic puppet fail [06:47:23] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.048 second response time [06:47:24] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.697 second response time [06:47:51] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:47:52] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Epic puppet fail [06:48:21] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:21] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Puppet has 2 failures [06:48:21] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Puppet has 4 failures [06:48:21] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:21] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:22] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:22] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 2 failures [06:48:22] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:22] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 2 failures [06:48:23] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:23] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:24] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:31] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:48:31] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:41] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 2 failures [06:48:42] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 1 failures [06:48:42] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Puppet has 2 failures [06:48:42] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 3 failures [06:48:54] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Puppet has 2 failures [06:48:55] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:02] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:02] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:11] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:16] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:16] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:16] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Epic puppet fail [06:49:16] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Puppet has 3 failures [06:49:16] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:16] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:16] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:21] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:21] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:21] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:21] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:22] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:31] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:31] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:31] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 3 failures [06:49:31] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:32] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 2 failures [06:49:32] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Puppet has 2 failures [06:49:32] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Puppet has 2 failures [06:49:32] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 4 failures [06:49:41] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:45] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:45] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:49:45] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:46] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:53] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:49:53] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:02] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:02] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:02] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Puppet has 3 failures [06:50:02] PROBLEM - puppet last run on db72 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:02] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:02] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:03] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 3 failures [06:50:03] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 3 failures [06:50:03] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:04] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:04] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:05] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:05] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:06] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:11] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 4 failures [06:50:11] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:11] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:11] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:50:12] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:12] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Puppet has 3 failures [06:50:12] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:12] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:12] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:13] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:13] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:14] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:14] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:15] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:15] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:24] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:25] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:25] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:25] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:25] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:25] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:25] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:26] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:26] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:31] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:31] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:31] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:31] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:31] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:31] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:32] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:32] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 1 failures [06:50:51] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:50:51] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:51:03] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:03] PROBLEM - puppet last run on db71 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:03] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:03] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:04] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 3 failures [06:51:04] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:04] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:04] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:04] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 3 failures [06:51:21] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 66 seconds ago with 0 failures [06:51:21] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:21] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 2 failures [06:51:22] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:22] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:51:22] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:22] PROBLEM - puppet last run on db74 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:22] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:23] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:23] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:24] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:31] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:51:32] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:32] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:51:41] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:41] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:42] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:51:42] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:51:42] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Puppet has 1 failures [06:51:42] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:52:01] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:02] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:11] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:21] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:52:22] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Puppet has 2 failures [06:52:22] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:22] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:52:22] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:52:22] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.417 second response time [06:52:31] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:31] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:52:43] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:45] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:46] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:51] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:52:51] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:51] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 3 failures [06:52:52] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:53] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Puppet has 1 failures [06:52:53] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:53:01] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:53:01] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:53:01] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:01] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:22] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:23] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Puppet has 2 failures [06:53:23] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:23] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:23] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:53:31] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:31] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:31] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:53:31] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:31] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:31] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:31] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:32] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:32] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:33] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:33] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:53:41] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Puppet has 2 failures [06:53:43] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:43] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 2 failures [06:53:44] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:44] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:51] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:11] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:54:21] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:54:21] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:54:22] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:54:25] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:25] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:54:25] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:54:25] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:25] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:54:25] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:54:31] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:41] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 1 failures [06:54:42] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:54:42] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:55:04] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 2 failures [06:55:04] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:04] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:55:04] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:55:04] RECOVERY - puppet last run on db72 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:55:04] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:55:11] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:11] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:55:21] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [06:55:21] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:55:21] RECOVERY - puppet last run on ms-be1007 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:55:21] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:55:21] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:55:21] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:55:22] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:22] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:55:22] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:55:31] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:55:31] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:55:31] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:55:32] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:55:42] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:55:42] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:55:42] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:55:42] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:55:42] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:55:43] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:43] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:51] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:56:01] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:56:02] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:56:02] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:56:02] RECOVERY - puppet last run on db71 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:56:02] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:56:02] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:56:02] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:56:22] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:56:24] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:56:24] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:56:24] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:56:24] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:56:25] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:56:25] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:56:25] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:56:25] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:56:41] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:56:41] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:56:42] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:56:42] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:57:13] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:57:14] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:57:14] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:57:21] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:57:22] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:57:22] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:57:32] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [06:58:11] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:58:22] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 1 failures [06:58:22] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:59:01] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:59:01] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:59:02] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:59:02] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:59:02] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:59:32] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:59:32] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:00:23] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:00:32] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:00:41] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:01:22] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:01:31] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:01:32] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [07:01:52] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:02:41] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:03:24] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:03:31] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [07:03:35] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:03:41] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:03:42] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:04:04] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:04:11] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:04:11] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:04:12] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:04:21] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:04:21] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:04:22] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:04:22] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:04:22] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:04:22] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:04:22] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:04:33] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:05:02] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:05:11] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:05:11] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:05:11] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:05:25] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:05:31] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:05:32] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:05:41] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:05:51] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:06:21] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:06:21] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:06:31] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:06:31] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:06:31] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:06:41] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:06:41] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:06:52] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:06:55] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:07:21] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:07:22] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:07:31] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:07:33] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:07:52] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:09:35] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:10:31] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:10:31] RECOVERY - puppet last run on db74 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [07:12:22] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:14:32] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:16:42] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 63 seconds ago with 0 failures [07:18:31] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:18:42] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:18:44] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:19:51] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:20:32] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:20:41] RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [07:21:11] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:21:34] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:21:54] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:22:15] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:22:43] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 64 seconds ago with 0 failures [07:22:43] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:22:43] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:22:45] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:23:34] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:24:13] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:24:44] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:25:04] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:26:34] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:26:54] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:27:05] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Epic puppet fail [07:27:43] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:28:03] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.035 second response time [07:28:33] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:28:45] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Epic puppet fail [07:28:46] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:29:16] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 8.936 second response time [07:29:45] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Epic puppet fail [07:29:45] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Epic puppet fail [07:29:45] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: Puppet has 1 failures [07:30:06] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: Epic puppet fail [07:30:23] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Epic puppet fail [07:30:36] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Epic puppet fail [07:30:37] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Epic puppet fail [07:30:37] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Epic puppet fail [07:30:37] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Epic puppet fail [07:30:54] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Epic puppet fail [07:30:55] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Epic puppet fail [07:31:03] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Epic puppet fail [07:31:34] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Epic puppet fail [07:31:37] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Epic puppet fail [07:31:44] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Puppet has 1 failures [07:31:53] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: Epic puppet fail [07:31:54] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Epic puppet fail [07:31:55] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Epic puppet fail [07:32:13] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: Epic puppet fail [07:32:14] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Epic puppet fail [07:32:24] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:54] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Epic puppet fail [07:33:05] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Epic puppet fail [07:33:13] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:33:23] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 1 failures [07:33:33] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:33:34] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Epic puppet fail [07:33:34] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:34:04] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 1 failures [07:34:04] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Epic puppet fail [07:34:06] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Puppet has 1 failures [07:34:14] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Puppet has 2 failures [07:34:44] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Puppet has 1 failures [07:35:23] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Epic puppet fail [07:35:24] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 1 failures [07:35:24] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.907 second response time [07:35:24] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 1 failures [07:35:34] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Epic puppet fail [07:35:55] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Epic puppet fail [07:35:55] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Epic puppet fail [07:36:13] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 1 failures [07:36:14] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Puppet has 2 failures [07:36:14] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures [07:36:15] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 2 failures [07:36:15] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 1 failures [07:36:23] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 3 failures [07:36:23] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 3 failures [07:36:24] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Puppet has 1 failures [07:36:24] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 1 failures [07:36:33] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:36:35] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Epic puppet fail [07:36:35] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:36:35] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 2 failures [07:36:35] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Epic puppet fail [07:36:35] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Puppet has 1 failures [07:36:43] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 1 failures [07:36:46] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [07:36:53] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 6 failures [07:36:54] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:37:03] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:37:16] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Epic puppet fail [07:37:27] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Epic puppet fail [07:37:35] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:37:35] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 3 failures [07:37:45] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Epic puppet fail [07:37:46] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Epic puppet fail [07:37:46] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Epic puppet fail [07:37:46] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 1 failures [07:37:46] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 1 failures [07:37:47] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Epic puppet fail [07:37:55] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 2 failures [07:37:55] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [07:37:55] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Puppet has 1 failures [07:37:56] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:37:56] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Epic puppet fail [07:37:56] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 1 failures [07:37:56] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 2 failures [07:37:56] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 2 failures [07:37:57] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Epic puppet fail [07:38:06] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:06] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:06] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 2 failures [07:38:16] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 2 failures [07:38:16] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Puppet has 5 failures [07:38:16] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 3 failures [07:38:17] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 4 failures [07:38:25] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:38:25] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:25] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:26] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:26] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet has 2 failures [07:38:26] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 2 failures [07:38:26] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:26] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 3 failures [07:38:27] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 4 failures [07:38:27] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:28] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:28] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:38:29] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 2 failures [07:38:48] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:38:49] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:49] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [07:38:50] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 2 failures [07:38:50] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 3 failures [07:39:05] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:06] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 2 failures [07:39:06] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Puppet has 2 failures [07:39:06] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:06] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:39:06] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Epic puppet fail [07:39:07] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Epic puppet fail [07:39:08] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 6 failures [07:39:09] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 3 failures [07:39:09] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 3 failures [07:39:09] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 4 failures [07:39:09] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 2 failures [07:39:09] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 2 failures [07:39:10] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:10] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:15] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:15] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 2 failures [07:39:15] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Puppet has 2 failures [07:39:16] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:25] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 3 failures [07:39:25] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 5 failures [07:39:25] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:26] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 4.940 second response time [07:39:26] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Puppet has 3 failures [07:39:45] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Epic puppet fail [07:39:49] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:49] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Epic puppet fail [07:39:49] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Epic puppet fail [07:39:49] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:49] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Epic puppet fail [07:39:49] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 1 failures [07:39:56] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:39:56] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:39:56] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 5 failures [07:39:56] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 4 failures [07:39:56] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Epic puppet fail [07:39:57] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Puppet has 3 failures [07:40:05] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Puppet has 3 failures [07:40:05] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:40:05] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 2 failures [07:40:05] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:40:06] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [07:40:16] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:40:16] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Puppet has 1 failures [07:40:16] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Puppet has 1 failures [07:40:16] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:40:25] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 2 failures [07:40:25] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Epic puppet fail [07:40:25] PROBLEM - check if dhclient is running on palladium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:26] PROBLEM - puppet last run on palladium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:26] PROBLEM - DPKG on palladium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:26] PROBLEM - Disk space on palladium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:26] PROBLEM - check configured eth on palladium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:26] PROBLEM - RAID on palladium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:27] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 1 failures [07:40:29] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Epic puppet fail [07:40:29] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:40:29] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Puppet has 1 failures [07:40:29] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:40:29] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 2 failures [07:40:30] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 3 failures [07:40:30] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 1 failures [07:40:45] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Epic puppet fail [07:40:48] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 1 failures [07:40:55] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Puppet has 3 failures [07:40:56] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 5 failures [07:40:56] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 5 failures [07:40:56] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Puppet has 1 failures [07:41:25] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Puppet has 2 failures [07:41:25] RECOVERY - check if dhclient is running on palladium is OK: PROCS OK: 0 processes with command name dhclient [07:41:25] RECOVERY - Disk space on palladium is OK: DISK OK [07:41:25] RECOVERY - DPKG on palladium is OK: All packages OK [07:41:25] RECOVERY - check configured eth on palladium is OK: NRPE: Unable to read output [07:41:25] RECOVERY - RAID on palladium is OK: RAID STATUS: OPTIMAL [07:41:25] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 70 seconds ago with 0 failures [07:41:26] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Puppet has 2 failures [07:41:26] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Puppet has 2 failures [07:41:28] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: Puppet has 1 failures [07:41:28] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 3 failures [07:41:30] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 1 failures [07:41:35] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:41:35] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 1 failures [07:41:35] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:41:46] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 1 failures [07:41:46] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 2 failures [07:41:46] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 2 failures [07:41:59] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Puppet has 2 failures [07:41:59] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 2 failures [07:42:07] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Puppet has 2 failures [07:42:07] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Puppet has 1 failures [07:42:07] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 1 failures [07:42:08] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 2 failures [07:42:15] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 1 failures [07:42:15] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Puppet has 2 failures [07:42:25] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 2 failures [07:42:35] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:35] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:42:35] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Puppet has 2 failures [07:42:57] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:06] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 2 failures [07:43:17] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Puppet has 2 failures [07:43:17] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:18] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:25] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:25] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:43:25] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:25] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:25] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:43:26] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:26] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:26] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:26] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:45] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 2 failures [07:43:45] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Puppet has 1 failures [07:43:46] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 2 failures [07:44:15] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Puppet has 1 failures [07:44:19] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 2 failures [07:44:20] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 1 failures [07:44:20] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Puppet has 1 failures [07:44:35] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 1.596 second response time [07:45:25] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:45:38] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Puppet has 3 failures [07:45:47] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:46:45] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:47:16] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:47:45] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:58] PROBLEM - puppetmaster https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:25] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:48:46] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 5.920 second response time [07:48:58] RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 9.612 second response time [07:49:06] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:49:17] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:49:35] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:49:57] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:50:14] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:50:14] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:50:45] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:50:46] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:50:46] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:50:58] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:51:05] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:51:15] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:51:25] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:51:25] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:51:26] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:51:26] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:51:35] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:51:45] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:51:45] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:51:45] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:51:45] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:51:46] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:52:28] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:52:35] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:52:45] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:52:45] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:52:57] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:53:27] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:53:36] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:53:45] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:53:45] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:53:45] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:53:45] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:53:57] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:54:02] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:54:04] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:54:15] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:54:15] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:54:25] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:54:25] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:54:26] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:54:26] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:54:26] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:54:26] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:54:26] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:54:26] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:54:35] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:54:39] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:54:40] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:54:40] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:54:40] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [07:54:40] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:54:40] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:54:44] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:54:44] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:54:55] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 63 seconds ago with 0 failures [07:54:55] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:54:55] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:54:56] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:54:56] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:54:56] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:55:07] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:55:07] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:55:07] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:55:08] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:55:08] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:55:15] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [07:55:15] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:55:16] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:55:16] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:55:16] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:55:25] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:55:31] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:55:31] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:55:31] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:55:32] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:55:32] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:55:32] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:55:32] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:55:33] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [07:55:33] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:55:34] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:55:34] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:55:35] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:55:35] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:55:36] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:55:36] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:55:37] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:55:37] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:55:38] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:55:38] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:55:39] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:55:39] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:55:40] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:55:41] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:55:41] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:55:41] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:55:42] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:55:42] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:55:43] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:55:43] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:55:44] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:55:44] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:55:45] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:55:45] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [07:55:46] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [07:55:46] RECOVERY - puppet last run on ms-be3001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:55:47] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:55:47] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:55:48] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:55:48] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:55:49] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:55:49] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:55:55] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:55:56] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:55:56] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:55:56] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:55:56] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:55:56] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:55:56] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:55:57] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:55:57] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [07:55:58] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:55:58] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:55:59] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [07:55:59] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:56:00] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:56:00] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:56:01] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:56:05] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:56:06] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:56:06] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:56:06] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:56:06] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 67 seconds ago with 0 failures [07:56:06] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:56:06] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:56:07] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:56:07] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:56:13] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:56:13] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:56:25] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:56:26] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:56:26] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:56:26] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:56:26] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:56:26] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:56:26] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:56:26] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:56:27] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:56:28] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:56:28] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:56:28] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:56:29] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:56:29] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:56:30] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:56:30] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:56:35] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:56:35] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:56:35] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:56:36] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:56:45] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:57:00] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:57:15] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [07:57:15] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:57:26] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:57:26] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:57:59] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:57:59] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:58:26] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:58:35] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:58:36] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:58:45] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:58:45] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:59:35] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:59:36] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:59:56] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [08:00:55] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:01:15] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:01:26] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:01:35] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [08:01:45] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [08:02:45] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [08:02:46] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [08:03:06] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [08:03:55] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [08:09:06] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 4.86515145561e-16 [08:22:51] I guess we should start putting a note with the rt person's tz (at 5 am I was not here, don't know what happened in the end) [08:42:36] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Epic puppet fail [09:01:57] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [09:35:58] PROBLEM - ElasticSearch health check for shards on logstash1003 is CRITICAL: CRITICAL - elasticsearch http://10.64.32.136:9200/_cluster/health error while fetching: Request timed out. [09:45:51] (03PS1) 10Springle: repool es1007, depool es1005 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/160271 [09:51:28] (03CR) 10Springle: [C: 032] repool es1007, depool es1005 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/160271 (owner: 10Springle) [09:51:32] (03Merged) 10jenkins-bot: repool es1007, depool es1005 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/160271 (owner: 10Springle) [09:56:04] !log springle Synchronized wmf-config/db-eqiad.php: repool es1007, depool es1005 (duration: 00m 10s) [09:56:09] Logged the message, Master [09:56:35] woo /srv/mediawiki-staging on tin... nice [10:20:45] (03PS1) 10Springle: prepare es1005 for upgrade [puppet] - 10https://gerrit.wikimedia.org/r/160274 [10:22:09] (03CR) 10Springle: [C: 032] prepare es1005 for upgrade [puppet] - 10https://gerrit.wikimedia.org/r/160274 (owner: 10Springle) [10:26:49] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 1 failures [10:34:49] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [10:37:30] !log restart es1005 [10:37:35] Logged the message, Master [10:45:24] (03PS1) 10Springle: promote es1006 to es2 master [dns] - 10https://gerrit.wikimedia.org/r/160275 [10:46:20] (03CR) 10Springle: [C: 032] promote es1006 to es2 master [dns] - 10https://gerrit.wikimedia.org/r/160275 (owner: 10Springle) [10:55:31] (03PS1) 10Springle: prepare es1005 for upgrade, part 2 [puppet] - 10https://gerrit.wikimedia.org/r/160276 [10:56:26] (03CR) 10Springle: [C: 032] prepare es1005 for upgrade, part 2 [puppet] - 10https://gerrit.wikimedia.org/r/160276 (owner: 10Springle) [11:06:52] (03PS1) 10Springle: Update topology for es2 master switch. Disable snapshots on es1007, temporarily switched to on db1004. [puppet] - 10https://gerrit.wikimedia.org/r/160277 [11:09:41] (03PS1) 10Springle: repool es1005, enable writes to es2 with es1006 as new master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/160278 [11:10:01] (03CR) 10Springle: [C: 032] Update topology for es2 master switch. Disable snapshots on es1007, temporarily switched to on db1004. [puppet] - 10https://gerrit.wikimedia.org/r/160277 (owner: 10Springle) [11:12:24] (03CR) 10Springle: [C: 032] repool es1005, enable writes to es2 with es1006 as new master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/160278 (owner: 10Springle) [11:12:28] (03Merged) 10jenkins-bot: repool es1005, enable writes to es2 with es1006 as new master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/160278 (owner: 10Springle) [11:14:08] !log springle Synchronized wmf-config/db-eqiad.php: repool es1005 (duration: 00m 07s) [11:14:14] Logged the message, Master [11:23:21] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:30:38] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:08] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Puppet has 1 failures [12:42:35] !log Ran sync-common on mw1053 to stop "Unrecognized job type 'ChangeNotification'." exceptions [12:42:39] Logged the message, Master [12:42:39] sjoerddebruin: ^ [12:42:45] :) [12:42:46] Should be fine now [12:43:25] It's still growing, hoo. [12:43:41] Really? [12:43:45] The exceptions stopped [12:44:09] https://nl.wikipedia.org/wiki/Blauwbekagamen showed up, Wikidata-edit on 43. [12:45:16] :S [12:45:28] And do you think the David ones need null edits? [12:45:47] yep, maybe purge via api with forcelinkupdates is enough [12:45:53] but null edits will work for sure [12:45:59] (just open action=edit and save) [12:46:17] 2014-09-14 12:43:30 mw1007 nlwiki: refreshLinks Blauwbekagamen rootJobSignature=... rootJobTimestamp=20140914124327 t=788 good [12:46:24] 26 pages... :/ [12:46:27] that's not even that host [12:46:35] (not an hhvm one AFAIK) [12:46:59] but that page has recently been moved [12:47:03] Yes. [12:47:28] * sjoerddebruin wish there was a null edit script [12:47:46] There surely is one... but I don't know of one offhand [12:48:09] Wikia has. http://dev.wikia.com/wiki/NullEditButton [12:48:59] http://dev.wikia.com/wiki/NullEditButton/code.js [12:49:12] CC-BY-SA [12:49:40] Yes, but it needs some adjustments. And I think it can be shorter. [12:50:00] yeah, probably [12:50:08] Hm https://en.wikipedia.org/wiki/User:Splarka/nulledit.js [12:50:21] 2008... [12:50:30] Going to test that. [12:50:35] mh [12:50:42] that looks like it could be exploited [12:50:47] to create pages [12:51:03] ? [12:51:05] * hoo wouldn't use that [12:51:09] Okay. :P [12:52:19] Will forcerecursivelinkupdate also work? [12:52:36] that option is evil [12:52:42] you mean on the special page? [12:53:18] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [12:53:32] Idk, it's mentioned on https://en.wikipedia.org/wiki/Wikipedia:Purge#Null_edit [12:58:39] sjoerddebruin: Still seeing new cases? [12:59:08] Ehm, not in the area of pages I've already checked. [13:01:36] (03PS1) 10Gage: Logstash: Hadoop: Grok patterns for Thread names [puppet] - 10https://gerrit.wikimedia.org/r/160281 [13:03:15] (03PS2) 10Gage: Logstash: Hadoop: Grok patterns for Thread names [puppet] - 10https://gerrit.wikimedia.org/r/160281 [13:04:11] (03CR) 10Gage: [C: 032] Logstash: Hadoop: Grok patterns for Thread names [puppet] - 10https://gerrit.wikimedia.org/r/160281 (owner: 10Gage) [13:06:19] :) [13:06:30] Now adjusting that Wikia-code. [13:06:33] Removing crap etc. :P [13:22:55] <_joe_> springle: thanks for the update on mw1053 [13:23:28] _joe_: Saw my sync-common on it earlier? [13:23:36] Not sure why exactly that fixes the issues... but it does [13:24:05] <_joe_> hoo: I think I know, but right no I'm disabling the JR there [13:24:10] <_joe_> until monday [13:24:19] Yeah, makes sense [13:24:23] <_joe_> !log stopped puppet aand the JR on mw1053 [13:24:26] that's not the first time this happened, btw [13:24:28] Logged the message, Master [13:24:48] <_joe_> :/ [13:24:57] <_joe_> I really gotta go now [13:25:20] cu :) [13:29:26] <_joe_> eh sorry, italian sunday meal with friends... I've already been unpolite enough to check email, rush here and do something about this :) [13:45:48] :S db1005 [15:03:08] (03PS1) 10Jackmcbarn: Set $wgAbuseFilterAnonBlockDuration to 3 months on mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/160286 (https://bugzilla.wikimedia.org/70828) [15:05:22] * hoo kicks reviewer-bot [15:05:39] works :D [17:18:11] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Epic puppet fail [17:25:23] hoo: No more issues btw. [17:26:53] :) [17:28:06] Only one, but that's a non-SUL user moving a page. :/ [17:28:30] Issues like that will be gone, soonish :) [17:29:20] well, mostly [17:29:24] blocked user still wont work [17:29:32] and users not having an account on WD.org [17:30:56] Hm. : [17:37:31] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:01:49] (03CR) 10Ori.livneh: [C: 032] hhvm: add comment about translation cache [puppet] - 10https://gerrit.wikimedia.org/r/160231 (owner: 10Ori.livneh) [20:04:52] (03CR) 10Ori.livneh: Clean up salt::grain (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/153783 (owner: 10Ori.livneh) [20:06:54] (03PS1) 10Gage: Logstash: Hadoop: more grok patterns for Thread [puppet] - 10https://gerrit.wikimedia.org/r/160296 [20:07:58] (03CR) 10Gage: [C: 032] Logstash: Hadoop: more grok patterns for Thread [puppet] - 10https://gerrit.wikimedia.org/r/160296 (owner: 10Gage) [20:11:06] (03CR) 10Ori.livneh: Clean up salt::minion (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/153727 (owner: 10Ori.livneh) [20:11:16] !log springle Synchronized wmf-config/db-eqiad.php: depool db1062, locked up (duration: 00m 09s) [20:11:22] Logged the message, Master [20:14:42] (03PS1) 10Gage: no erb comments in files! [puppet] - 10https://gerrit.wikimedia.org/r/160297 [20:15:41] (03CR) 10Gage: [C: 032] no erb comments in files! [puppet] - 10https://gerrit.wikimedia.org/r/160297 (owner: 10Gage) [20:27:48] (03PS1) 10Ori.livneh: Increase maximum number of open files for HHVM from 1024 to 4096 [puppet] - 10https://gerrit.wikimedia.org/r/160298 (https://bugzilla.wikimedia.org/70825) [20:29:00] (03CR) 10Giuseppe Lavagetto: [C: 031] Increase maximum number of open files for HHVM from 1024 to 4096 [puppet] - 10https://gerrit.wikimedia.org/r/160298 (https://bugzilla.wikimedia.org/70825) (owner: 10Ori.livneh) [20:33:57] (03PS2) 10Giuseppe Lavagetto: Increase maximum number of open files for HHVM from 1024 to 4096 [puppet] - 10https://gerrit.wikimedia.org/r/160298 (https://bugzilla.wikimedia.org/70825) (owner: 10Ori.livneh) [20:34:48] (03CR) 10Giuseppe Lavagetto: [C: 032] Increase maximum number of open files for HHVM from 1024 to 4096 [puppet] - 10https://gerrit.wikimedia.org/r/160298 (https://bugzilla.wikimedia.org/70825) (owner: 10Ori.livneh) [20:35:12] thanks [20:36:48] (03PS3) 10Ori.livneh: Clean up salt::grain [puppet] - 10https://gerrit.wikimedia.org/r/153783 [20:36:50] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Puppet last ran 26771 seconds ago, expected 14400 [20:36:56] (03CR) 10Ori.livneh: [C: 032 V: 032] Clean up salt::grain [puppet] - 10https://gerrit.wikimedia.org/r/153783 (owner: 10Ori.livneh) [20:37:28] !log enabling puppet on mw1053 [20:37:31] Logged the message, Master [20:38:37] <_joe_> ori_: I already did [20:38:52] ok cool [20:38:55] <_joe_> I was about to log that :) [20:39:02] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:39:02] (03PS11) 10Ori.livneh: Clean up salt::minion [puppet] - 10https://gerrit.wikimedia.org/r/153727 [20:39:09] (03CR) 10Ori.livneh: [C: 032 V: 032] Clean up salt::minion [puppet] - 10https://gerrit.wikimedia.org/r/153727 (owner: 10Ori.livneh) [20:41:40] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Epic puppet fail [20:42:02] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: Epic puppet fail [20:42:11] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Epic puppet fail [20:42:11] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Epic puppet fail [20:49:52] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Epic puppet fail [21:04:45] (03PS1) 10Ori.livneh: Fix-up for param name and leading whitespace in Id158f154 [puppet] - 10https://gerrit.wikimedia.org/r/160299 [21:05:33] (03CR) 10Ori.livneh: [C: 032] Fix-up for param name and leading whitespace in Id158f154 [puppet] - 10https://gerrit.wikimedia.org/r/160299 (owner: 10Ori.livneh) [21:09:31] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Epic puppet fail [21:11:36] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Epic puppet fail [21:11:36] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Epic puppet fail [21:11:36] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Epic puppet fail [21:11:47] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Epic puppet fail [21:11:47] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Epic puppet fail [21:11:55] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: Epic puppet fail [21:11:56] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Epic puppet fail [21:11:56] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: Epic puppet fail [21:11:56] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: Epic puppet fail [21:12:05] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Epic puppet fail [21:12:06] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: Epic puppet fail [21:12:06] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Epic puppet fail [21:12:15] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Epic puppet fail [21:12:15] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: Epic puppet fail [21:12:20] PROBLEM - puppet last run on tarin is CRITICAL: CRITICAL: Epic puppet fail [21:12:33] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Epic puppet fail [21:12:33] PROBLEM - puppet last run on mw1062 is CRITICAL: CRITICAL: Epic puppet fail [21:12:43] ok, that was my change. reverting. sorry. [21:12:43] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Epic puppet fail [21:12:43] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Epic puppet fail [21:12:44] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Epic puppet fail [21:12:45] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: Epic puppet fail [21:12:45] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Epic puppet fail [21:12:45] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Epic puppet fail [21:12:45] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: Epic puppet fail [21:12:45] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Epic puppet fail [21:12:45] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Epic puppet fail [21:12:46] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: Epic puppet fail [21:12:49] <_joe_> (In yoda's voice) use the compiler, ori! [21:12:51] <_joe_> :) [21:12:53] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Epic puppet fail [21:12:53] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Epic puppet fail [21:12:53] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Epic puppet fail [21:12:53] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Epic puppet fail [21:12:53] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Epic puppet fail [21:12:53] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Epic puppet fail [21:12:53] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Epic puppet fail [21:12:54] PROBLEM - puppet last run on mw1038 is CRITICAL: CRITICAL: Epic puppet fail [21:12:54] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Epic puppet fail [21:12:55] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Epic puppet fail [21:13:03] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Epic puppet fail [21:13:03] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: Epic puppet fail [21:13:03] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Epic puppet fail [21:13:03] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Epic puppet fail [21:13:04] PROBLEM - puppet last run on search1020 is CRITICAL: CRITICAL: Epic puppet fail [21:13:04] <_joe_> it's just noise [21:13:04] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Epic puppet fail [21:13:04] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Epic puppet fail [21:13:07] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Epic puppet fail [21:13:07] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: Epic puppet fail [21:13:13] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Epic puppet fail [21:13:13] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Epic puppet fail [21:13:14] PROBLEM - puppet last run on dbstore1001 is CRITICAL: CRITICAL: Epic puppet fail [21:13:14] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Epic puppet fail [21:13:14] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Epic puppet fail [21:13:14] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Epic puppet fail [21:13:14] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Epic puppet fail [21:13:23] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: Epic puppet fail [21:13:23] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Epic puppet fail [21:13:23] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Epic puppet fail [21:13:23] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Epic puppet fail [21:13:23] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Epic puppet fail [21:13:24] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Epic puppet fail [21:13:24] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Epic puppet fail [21:13:24] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Epic puppet fail [21:13:24] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Epic puppet fail [21:13:25] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Epic puppet fail [21:13:44] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Epic puppet fail [21:13:44] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Epic puppet fail [21:13:45] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: Epic puppet fail [21:13:45] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Epic puppet fail [21:13:45] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Epic puppet fail [21:13:55] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Epic puppet fail [21:13:56] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Epic puppet fail [21:13:56] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Epic puppet fail [21:13:56] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Epic puppet fail [21:13:56] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Epic puppet fail [21:13:56] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Epic puppet fail [21:13:56] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Epic puppet fail [21:14:06] PROBLEM - puppet last run on db73 is CRITICAL: CRITICAL: Epic puppet fail [21:14:06] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Epic puppet fail [21:14:06] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Epic puppet fail [21:14:07] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Epic puppet fail [21:14:07] PROBLEM - puppet last run on db1017 is CRITICAL: CRITICAL: Epic puppet fail [21:14:07] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Epic puppet fail [21:14:07] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Epic puppet fail [21:14:07] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Epic puppet fail [21:14:08] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Epic puppet fail [21:14:08] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Epic puppet fail [21:14:13] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Epic puppet fail [21:14:13] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Epic puppet fail [21:14:13] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Epic puppet fail [21:14:14] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Epic puppet fail [21:14:14] PROBLEM - puppet last run on mw1200 is CRITICAL: CRITICAL: Epic puppet fail [21:14:14] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Epic puppet fail [21:14:14] PROBLEM - puppet last run on analytics1040 is CRITICAL: CRITICAL: Epic puppet fail [21:14:14] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Epic puppet fail [21:14:14] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Epic puppet fail [21:14:15] PROBLEM - puppet last run on es1008 is CRITICAL: CRITICAL: Epic puppet fail [21:14:16] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Epic puppet fail [21:14:21] can you kick the bot for a min? [21:14:24] PROBLEM - puppet last run on ssl1002 is CRITICAL: CRITICAL: Epic puppet fail [21:14:24] PROBLEM - puppet last run on analytics1020 is CRITICAL: CRITICAL: Epic puppet fail [21:14:24] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Epic puppet fail [21:14:24] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Epic puppet fail [21:14:43] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Epic puppet fail [21:14:43] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Epic puppet fail [21:14:43] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Epic puppet fail [21:14:43] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: Epic puppet fail [21:14:44] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Epic puppet fail [21:14:44] PROBLEM - puppet last run on gold is CRITICAL: CRITICAL: Epic puppet fail [21:14:44] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Epic puppet fail [21:14:54] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Epic puppet fail [21:14:55] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Epic puppet fail [21:14:55] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Epic puppet fail [21:14:55] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Epic puppet fail [21:14:55] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Epic puppet fail [21:14:55] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Epic puppet fail [21:14:55] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Epic puppet fail [21:14:55] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Epic puppet fail [21:15:03] PROBLEM - puppet last run on analytics1041 is CRITICAL: CRITICAL: Epic puppet fail [21:15:03] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Epic puppet fail [21:15:03] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Epic puppet fail [21:15:04] PROBLEM - puppet last run on rbf1002 is CRITICAL: CRITICAL: Epic puppet fail [21:15:04] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Epic puppet fail [21:15:04] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Epic puppet fail [21:15:04] PROBLEM - puppet last run on platinum is CRITICAL: CRITICAL: Epic puppet fail [21:15:04] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Epic puppet fail [21:15:05] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Epic puppet fail [21:15:13] PROBLEM - puppet last run on db74 is CRITICAL: CRITICAL: Epic puppet fail [21:15:14] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Epic puppet fail [21:15:14] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Epic puppet fail [21:15:14] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Epic puppet fail [21:15:14] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Epic puppet fail [21:15:14] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: Epic puppet fail [21:15:14] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Epic puppet fail [21:15:14] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Epic puppet fail [21:15:15] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: Epic puppet fail [21:15:15] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Epic puppet fail [21:15:16] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Epic puppet fail [21:15:16] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Epic puppet fail [21:15:44] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Epic puppet fail [21:15:44] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Epic puppet fail [21:15:44] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Epic puppet fail [21:15:44] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Epic puppet fail [21:15:44] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Epic puppet fail [21:15:44] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Epic puppet fail [21:15:45] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Epic puppet fail [21:15:45] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Epic puppet fail [21:15:54] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Epic puppet fail [21:15:55] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Epic puppet fail [21:15:55] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Epic puppet fail [21:15:55] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Epic puppet fail [21:15:55] PROBLEM - puppet last run on mw1176 is CRITICAL: CRITICAL: Epic puppet fail [21:15:55] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Epic puppet fail [21:15:55] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Epic puppet fail [21:16:03] PROBLEM - puppet last run on analytics1035 is CRITICAL: CRITICAL: Epic puppet fail [21:16:09] PROBLEM - puppet last run on mw1205 is CRITICAL: CRITICAL: Epic puppet fail [21:16:10] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Epic puppet fail [21:16:10] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Epic puppet fail [21:16:27] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Epic puppet fail [21:16:27] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Epic puppet fail [21:16:27] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Epic puppet fail [21:16:27] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Epic puppet fail [21:16:27] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Epic puppet fail [21:16:27] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Epic puppet fail [21:16:27] PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Epic puppet fail [21:16:28] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Epic puppet fail [21:16:29] PROBLEM - puppet last run on sca1002 is CRITICAL: CRITICAL: Epic puppet fail [21:16:37] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Epic puppet fail [21:16:37] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Epic puppet fail [21:16:37] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Epic puppet fail [21:16:38] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Epic puppet fail [21:16:38] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Epic puppet fail [21:16:38] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Epic puppet fail [21:16:38] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Epic puppet fail [21:16:38] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Epic puppet fail [21:16:38] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Epic puppet fail [21:16:39] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Epic puppet fail [21:16:39] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Epic puppet fail [21:16:40] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Epic puppet fail [21:16:41] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Epic puppet fail [21:16:47] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Epic puppet fail [21:16:55] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Epic puppet fail [21:16:55] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Epic puppet fail [21:16:55] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Epic puppet fail [21:16:58] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Epic puppet fail [21:17:08] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Epic puppet fail [21:17:09] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Epic puppet fail [21:17:09] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Epic puppet fail [21:17:09] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Epic puppet fail [21:17:10] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Epic puppet fail [21:17:10] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Epic puppet fail [21:17:10] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Epic puppet fail [21:17:10] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Epic puppet fail [21:17:10] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Epic puppet fail [21:17:10] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Epic puppet fail [21:17:11] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Epic puppet fail [21:17:18] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Epic puppet fail [21:17:18] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Epic puppet fail [21:17:19] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Epic puppet fail [21:17:19] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Epic puppet fail [21:17:19] PROBLEM - ElasticSearch health check on logstash1001 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.138 [21:17:38] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Epic puppet fail [21:17:48] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Epic puppet fail [21:17:51] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Epic puppet fail [21:17:51] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Epic puppet fail [21:17:58] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Epic puppet fail [21:17:58] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: Epic puppet fail [21:17:58] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Epic puppet fail [21:17:59] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: Epic puppet fail [21:17:59] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Epic puppet fail [21:17:59] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Epic puppet fail [21:17:59] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Epic puppet fail [21:17:59] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Epic puppet fail [21:18:00] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Epic puppet fail [21:18:00] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Epic puppet fail [21:18:08] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Epic puppet fail [21:18:08] PROBLEM - ElasticSearch health check for shards on logstash1001 is CRITICAL: CRITICAL - elasticsearch http://10.64.32.138:9200/_cluster/health error while fetching: Request timed out. [21:18:08] PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Epic puppet fail [21:18:09] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Epic puppet fail [21:18:09] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Epic puppet fail [21:18:09] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Epic puppet fail [21:18:09] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Epic puppet fail [21:18:09] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: Epic puppet fail [21:18:10] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Epic puppet fail [21:18:10] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Epic puppet fail [21:18:11] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Epic puppet fail [21:18:18] PROBLEM - puppet last run on logstash1002 is CRITICAL: CRITICAL: Epic puppet fail [21:18:19] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Epic puppet fail [21:18:19] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Epic puppet fail [21:18:19] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Epic puppet fail [21:18:19] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Epic puppet fail [21:18:19] PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Epic puppet fail [21:18:19] PROBLEM - puppet last run on mw1129 is CRITICAL: CRITICAL: Epic puppet fail [21:18:20] PROBLEM - puppet last run on mw1002 is CRITICAL: CRITICAL: Epic puppet fail [21:18:20] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Epic puppet fail [21:18:20] PROBLEM - puppet last run on polonium is CRITICAL: CRITICAL: Epic puppet fail [21:18:28] PROBLEM - ElasticSearch health check for shards on logstash1002 is CRITICAL: CRITICAL - elasticsearch http://10.64.32.137:9200/_cluster/health error while fetching: Request timed out. [21:18:31] PROBLEM - ElasticSearch health check on logstash1002 is CRITICAL: CRITICAL - Could not connect to server 10.64.32.137 [21:18:40] PROBLEM - puppet last run on virt1003 is CRITICAL: CRITICAL: Epic puppet fail [21:18:41] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Epic puppet fail [21:18:41] PROBLEM - puppet last run on virt1004 is CRITICAL: CRITICAL: Epic puppet fail [21:18:41] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Epic puppet fail [21:18:41] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Epic puppet fail [21:18:41] PROBLEM - puppet last run on silver is CRITICAL: CRITICAL: Epic puppet fail [21:18:42] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Epic puppet fail [21:18:42] PROBLEM - puppet last run on mw1044 is CRITICAL: CRITICAL: Epic puppet fail [21:18:48] PROBLEM - puppet last run on cp4001 is CRITICAL: CRITICAL: Epic puppet fail [21:18:48] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Epic puppet fail [21:18:49] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Epic puppet fail [21:18:49] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Epic puppet fail [21:18:49] PROBLEM - puppet last run on db1052 is CRITICAL: CRITICAL: Epic puppet fail [21:18:49] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Epic puppet fail [21:18:49] PROBLEM - puppet last run on mw1206 is CRITICAL: CRITICAL: Epic puppet fail [21:18:49] PROBLEM - puppet last run on mw1076 is CRITICAL: CRITICAL: Epic puppet fail [21:18:50] PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Epic puppet fail [21:18:58] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: Epic puppet fail [21:18:58] PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Epic puppet fail [21:18:58] PROBLEM - puppet last run on virt1001 is CRITICAL: CRITICAL: Epic puppet fail [21:18:58] PROBLEM - puppet last run on cp4019 is CRITICAL: CRITICAL: Epic puppet fail [21:18:58] PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Epic puppet fail [21:18:59] PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Epic puppet fail [21:18:59] PROBLEM - puppet last run on antimony is CRITICAL: CRITICAL: Epic puppet fail [21:18:59] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Epic puppet fail [21:19:08] PROBLEM - puppet last run on analytics1016 is CRITICAL: CRITICAL: Epic puppet fail [21:19:08] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Epic puppet fail [21:19:08] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Epic puppet fail [21:19:09] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Epic puppet fail [21:19:13] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Epic puppet fail [21:19:14] PROBLEM - puppet last run on cp1058 is CRITICAL: CRITICAL: Epic puppet fail [21:19:14] PROBLEM - puppet last run on mw1125 is CRITICAL: CRITICAL: Epic puppet fail [21:19:14] PROBLEM - puppet last run on analytics1038 is CRITICAL: CRITICAL: Epic puppet fail [21:19:14] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Epic puppet fail [21:19:14] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Epic puppet fail [21:19:14] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Epic puppet fail [21:19:14] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: Epic puppet fail [21:19:16] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Epic puppet fail [21:19:16] PROBLEM - puppet last run on pc1002 is CRITICAL: CRITICAL: Epic puppet fail [21:19:16] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Epic puppet fail [21:19:17] PROBLEM - ElasticSearch health check for shards on logstash1003 is CRITICAL: CRITICAL - elasticsearch inactive shards 36 threshold =0.1% breach: {ustatus: uyellow, unumber_of_nodes: 2, uunassigned_shards: 31, utimed_out: False, uactive_primary_shards: 36, ucluster_name: uproduction-logstash-eqiad, urelocating_shards: 0, uactive_shards: 67, uinitializing_shards: 5, unumber_of_data_nodes: 2} [21:19:17] PROBLEM - puppet last run on ssl1005 is CRITICAL: CRITICAL: Epic puppet fail [21:19:18] PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Epic puppet fail [21:19:18] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Epic puppet fail [21:19:19] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Epic puppet fail [21:19:19] PROBLEM - puppet last run on cp1046 is CRITICAL: CRITICAL: Epic puppet fail [21:19:20] PROBLEM - puppet last run on mw1180 is CRITICAL: CRITICAL: Epic puppet fail [21:19:20] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Epic puppet fail [21:19:28] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: Epic puppet fail [21:19:28] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Epic puppet fail [21:19:28] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Epic puppet fail [21:19:29] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Epic puppet fail [21:19:29] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Epic puppet fail [21:19:29] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Epic puppet fail [21:19:38] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: Epic puppet fail [21:19:38] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Epic puppet fail [21:19:39] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: Epic puppet fail [21:19:39] PROBLEM - puppet last run on mw1049 is CRITICAL: CRITICAL: Epic puppet fail [21:19:39] PROBLEM - puppet last run on lvs4003 is CRITICAL: CRITICAL: Epic puppet fail [21:19:48] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Epic puppet fail [21:19:48] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: Epic puppet fail [21:19:52] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Epic puppet fail [21:19:53] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: Epic puppet fail [21:19:57] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Epic puppet fail [21:19:57] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Epic puppet fail [21:19:58] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Epic puppet fail [21:20:10] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: Epic puppet fail [21:20:10] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Epic puppet fail [21:20:10] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Epic puppet fail [21:20:10] PROBLEM - puppet last run on mw1111 is CRITICAL: CRITICAL: Epic puppet fail [21:20:11] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: Epic puppet fail [21:20:11] PROBLEM - puppet last run on elastic1006 is CRITICAL: CRITICAL: Epic puppet fail [21:20:11] PROBLEM - puppet last run on cp1050 is CRITICAL: CRITICAL: Epic puppet fail [21:20:12] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Epic puppet fail [21:20:12] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Epic puppet fail [21:20:12] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Epic puppet fail [21:20:13] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Epic puppet fail [21:20:13] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Epic puppet fail [21:20:14] PROBLEM - puppet last run on amssq42 is CRITICAL: CRITICAL: Epic puppet fail [21:20:18] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: Epic puppet fail [21:20:18] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Epic puppet fail [21:20:18] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: Epic puppet fail [21:20:18] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Epic puppet fail [21:20:18] PROBLEM - puppet last run on ssl1009 is CRITICAL: CRITICAL: Epic puppet fail [21:20:18] PROBLEM - puppet last run on analytics1022 is CRITICAL: CRITICAL: Epic puppet fail [21:20:19] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Epic puppet fail [21:20:19] PROBLEM - puppet last run on mw1030 is CRITICAL: CRITICAL: Epic puppet fail [21:20:19] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Epic puppet fail [21:20:20] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Epic puppet fail [21:20:21] PROBLEM - puppet last run on thallium is CRITICAL: CRITICAL: Epic puppet fail [21:20:21] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: Epic puppet fail [21:20:29] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Epic puppet fail [21:20:29] PROBLEM - puppet last run on db69 is CRITICAL: CRITICAL: Epic puppet fail [21:20:29] PROBLEM - puppet last run on db72 is CRITICAL: CRITICAL: Epic puppet fail [21:20:29] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Epic puppet fail [21:20:29] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: Epic puppet fail [21:20:30] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Epic puppet fail [21:20:30] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: Epic puppet fail [21:20:30] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Epic puppet fail [21:20:31] PROBLEM - puppet last run on ms-be1012 is CRITICAL: CRITICAL: Epic puppet fail [21:20:31] PROBLEM - puppet last run on rubidium is CRITICAL: CRITICAL: Epic puppet fail [21:20:32] PROBLEM - puppet last run on db1069 is CRITICAL: CRITICAL: Epic puppet fail [21:20:32] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Epic puppet fail [21:20:33] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: Epic puppet fail [21:20:33] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Epic puppet fail [21:20:49] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Epic puppet fail [21:20:49] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Epic puppet fail [21:20:50] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Epic puppet fail [21:20:50] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: Epic puppet fail [21:20:50] PROBLEM - puppet last run on mw1057 is CRITICAL: CRITICAL: Epic puppet fail [21:20:59] PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: Epic puppet fail [21:20:59] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Epic puppet fail [21:21:09] PROBLEM - puppet last run on mw1056 is CRITICAL: CRITICAL: Epic puppet fail [21:21:09] PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Epic puppet fail [21:21:09] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Epic puppet fail [21:21:09] PROBLEM - puppet last run on mw1146 is CRITICAL: CRITICAL: Epic puppet fail [21:21:10] PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Epic puppet fail [21:21:10] PROBLEM - puppet last run on wtp1018 is CRITICAL: CRITICAL: Epic puppet fail [21:21:10] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Epic puppet fail [21:21:10] PROBLEM - puppet last run on analytics1023 is CRITICAL: CRITICAL: Epic puppet fail [21:21:10] PROBLEM - puppet last run on db1004 is CRITICAL: CRITICAL: Epic puppet fail [21:21:11] PROBLEM - puppet last run on db1071 is CRITICAL: CRITICAL: Epic puppet fail [21:21:11] PROBLEM - puppet last run on amssq41 is CRITICAL: CRITICAL: Epic puppet fail [21:21:12] PROBLEM - puppet last run on db1055 is CRITICAL: CRITICAL: Epic puppet fail [21:21:13] PROBLEM - puppet last run on wtp1002 is CRITICAL: CRITICAL: Epic puppet fail [21:21:13] PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Epic puppet fail [21:21:14] PROBLEM - puppet last run on ms-be1007 is CRITICAL: CRITICAL: Epic puppet fail [21:21:19] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Epic puppet fail [21:21:33] PROBLEM - puppet last run on ms-be1009 is CRITICAL: CRITICAL: Epic puppet fail [21:21:33] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Epic puppet fail [21:21:34] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: Epic puppet fail [21:21:34] PROBLEM - puppet last run on mw1171 is CRITICAL: CRITICAL: Epic puppet fail [21:21:49] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Epic puppet fail [21:21:50] PROBLEM - puppet last run on mw1188 is CRITICAL: CRITICAL: Epic puppet fail [21:21:50] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Epic puppet fail [21:21:50] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Epic puppet fail [21:21:50] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Epic puppet fail [21:21:51] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Epic puppet fail [21:21:51] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Epic puppet fail [21:21:51] PROBLEM - puppet last run on search1024 is CRITICAL: CRITICAL: Epic puppet fail [21:21:59] PROBLEM - puppet last run on wtp1023 is CRITICAL: CRITICAL: Epic puppet fail [21:21:59] PROBLEM - puppet last run on acamar is CRITICAL: CRITICAL: Epic puppet fail [21:21:59] PROBLEM - puppet last run on wtp1022 is CRITICAL: CRITICAL: Epic puppet fail [21:21:59] PROBLEM - puppet last run on lvs1001 is CRITICAL: CRITICAL: Epic puppet fail [21:22:00] PROBLEM - puppet last run on wtp1004 is CRITICAL: CRITICAL: Epic puppet fail [21:22:00] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Epic puppet fail [21:22:09] PROBLEM - puppet last run on mw1053 is CRITICAL: CRITICAL: Epic puppet fail [21:22:09] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Epic puppet fail [21:22:10] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Epic puppet fail [21:22:10] PROBLEM - puppet last run on db1062 is CRITICAL: CRITICAL: Epic puppet fail [21:22:10] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: Epic puppet fail [21:22:10] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Epic puppet fail [21:22:10] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: Epic puppet fail [21:22:11] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Epic puppet fail [21:22:11] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: Epic puppet fail [21:22:11] PROBLEM - puppet last run on wtp1015 is CRITICAL: CRITICAL: Epic puppet fail [21:22:12] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: Epic puppet fail [21:22:12] PROBLEM - puppet last run on titanium is CRITICAL: CRITICAL: Epic puppet fail [21:22:19] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: Epic puppet fail [21:22:22] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Epic puppet fail [21:22:22] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: Epic puppet fail [21:22:22] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Epic puppet fail [21:22:22] PROBLEM - puppet last run on mc1007 is CRITICAL: CRITICAL: Epic puppet fail [21:22:22] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Epic puppet fail [21:22:23] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: Epic puppet fail [21:22:23] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: Epic puppet fail [21:22:24] PROBLEM - puppet last run on rcs1001 is CRITICAL: CRITICAL: Epic puppet fail [21:22:24] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: Epic puppet fail [21:22:25] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: Epic puppet fail [21:22:25] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Epic puppet fail [21:22:25] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: Epic puppet fail [21:22:26] PROBLEM - puppet last run on amssq38 is CRITICAL: CRITICAL: Epic puppet fail [21:22:27] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: Epic puppet fail [21:22:27] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: Epic puppet fail [21:22:27] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: Epic puppet fail [21:22:29] PROBLEM - puppet last run on es1002 is CRITICAL: CRITICAL: Epic puppet fail [21:22:29] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Epic puppet fail [21:22:29] PROBLEM - puppet last run on rdb1002 is CRITICAL: CRITICAL: Epic puppet fail [21:22:30] PROBLEM - puppet last run on db1001 is CRITICAL: CRITICAL: Epic puppet fail [21:22:30] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Epic puppet fail [21:22:59] PROBLEM - puppet last run on ssl1008 is CRITICAL: CRITICAL: Epic puppet fail [21:23:11] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Epic puppet fail [21:23:21] PROBLEM - puppet last run on analytics1011 is CRITICAL: CRITICAL: Epic puppet fail [21:23:22] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Epic puppet fail [21:23:22] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: Epic puppet fail [21:23:22] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Epic puppet fail [21:23:22] PROBLEM - puppet last run on mw1022 is CRITICAL: CRITICAL: Epic puppet fail [21:23:22] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: Epic puppet fail [21:23:22] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: Epic puppet fail [21:23:31] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Epic puppet fail [21:23:31] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: Epic puppet fail [21:23:31] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Epic puppet fail [21:23:32] PROBLEM - puppet last run on wtp1007 is CRITICAL: CRITICAL: Epic puppet fail [21:23:32] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Epic puppet fail [21:23:33] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: Epic puppet fail [21:23:33] PROBLEM - puppet last run on rdb1003 is CRITICAL: CRITICAL: Epic puppet fail [21:23:52] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Epic puppet fail [21:23:53] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Epic puppet fail [21:24:01] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Epic puppet fail [21:24:01] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Epic puppet fail [21:24:01] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Epic puppet fail [21:24:01] PROBLEM - puppet last run on mw1122 is CRITICAL: CRITICAL: Epic puppet fail [21:24:01] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Epic puppet fail [21:24:01] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Epic puppet fail [21:24:02] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Epic puppet fail [21:24:02] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: Epic puppet fail [21:24:03] PROBLEM - puppet last run on search1013 is CRITICAL: CRITICAL: Epic puppet fail [21:24:03] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: Epic puppet fail [21:24:03] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: Epic puppet fail [21:24:04] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Epic puppet fail [21:24:05] PROBLEM - puppet last run on ssl1007 is CRITICAL: CRITICAL: Epic puppet fail [21:24:22] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Epic puppet fail [21:24:22] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Epic puppet fail [21:24:22] PROBLEM - puppet last run on db1044 is CRITICAL: CRITICAL: Epic puppet fail [21:24:23] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: Epic puppet fail [21:24:23] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Epic puppet fail [21:24:23] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Epic puppet fail [21:24:23] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Epic puppet fail [21:24:23] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Epic puppet fail [21:24:24] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Epic puppet fail [21:24:24] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Epic puppet fail [21:24:32] PROBLEM - puppet last run on cp1037 is CRITICAL: CRITICAL: Epic puppet fail [21:24:32] PROBLEM - puppet last run on mw1139 is CRITICAL: CRITICAL: Epic puppet fail [21:24:33] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Epic puppet fail [21:24:36] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Epic puppet fail [21:24:41] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Epic puppet fail [21:24:41] PROBLEM - puppet last run on mercury is CRITICAL: CRITICAL: Epic puppet fail [21:24:41] PROBLEM - puppet last run on dbstore1002 is CRITICAL: CRITICAL: Epic puppet fail [21:24:42] PROBLEM - puppet last run on iodine is CRITICAL: CRITICAL: Epic puppet fail [21:24:42] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Epic puppet fail [21:24:42] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [21:24:42] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Epic puppet fail [21:24:42] PROBLEM - puppet last run on search1006 is CRITICAL: CRITICAL: Epic puppet fail [21:24:42] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Epic puppet fail [21:24:51] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Epic puppet fail [21:24:51] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Epic puppet fail [21:24:52] PROBLEM - puppet last run on calcium is CRITICAL: CRITICAL: Epic puppet fail [21:25:01] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Epic puppet fail [21:25:01] PROBLEM - puppet last run on analytics1027 is CRITICAL: CRITICAL: Epic puppet fail [21:25:01] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Epic puppet fail [21:25:01] PROBLEM - puppet last run on iridium is CRITICAL: CRITICAL: Epic puppet fail [21:25:01] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Epic puppet fail [21:25:01] PROBLEM - puppet last run on analytics1004 is CRITICAL: CRITICAL: Epic puppet fail [21:25:02] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Epic puppet fail [21:25:02] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Epic puppet fail [21:25:02] PROBLEM - puppet last run on search1022 is CRITICAL: CRITICAL: Epic puppet fail [21:25:03] PROBLEM - puppet last run on mw1093 is CRITICAL: CRITICAL: Epic puppet fail [21:25:04] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Epic puppet fail [21:25:04] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Epic puppet fail [21:25:05] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Epic puppet fail [21:25:23] PROBLEM - puppet last run on mw1193 is CRITICAL: CRITICAL: Epic puppet fail [21:25:23] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Epic puppet fail [21:25:23] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Epic puppet fail [21:25:23] PROBLEM - puppet last run on mw1215 is CRITICAL: CRITICAL: Epic puppet fail [21:25:23] PROBLEM - puppet last run on mw1107 is CRITICAL: CRITICAL: Epic puppet fail [21:25:23] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Epic puppet fail [21:25:23] PROBLEM - puppet last run on wtp1010 is CRITICAL: CRITICAL: Epic puppet fail [21:25:24] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Epic puppet fail [21:25:24] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Epic puppet fail [21:25:31] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Epic puppet fail [21:25:31] PROBLEM - puppet last run on ssl3001 is CRITICAL: CRITICAL: Epic puppet fail [21:25:31] PROBLEM - puppet last run on mw1086 is CRITICAL: CRITICAL: Epic puppet fail [21:25:31] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Epic puppet fail [21:25:31] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Epic puppet fail [21:25:32] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Epic puppet fail [21:25:32] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Epic puppet fail [21:25:32] PROBLEM - puppet last run on search1012 is CRITICAL: CRITICAL: Epic puppet fail [21:25:32] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: Epic puppet fail [21:25:33] PROBLEM - puppet last run on mw1204 is CRITICAL: CRITICAL: Epic puppet fail [21:25:33] PROBLEM - puppet last run on strontium is CRITICAL: CRITICAL: Epic puppet fail [21:25:34] PROBLEM - puppet last run on mw1112 is CRITICAL: CRITICAL: Epic puppet fail [21:25:41] PROBLEM - puppet last run on bast1001 is CRITICAL: CRITICAL: Epic puppet fail [21:25:41] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Epic puppet fail [21:25:41] PROBLEM - puppet last run on ms-be1002 is CRITICAL: CRITICAL: Epic puppet fail [21:25:41] PROBLEM - puppet last run on db60 is CRITICAL: CRITICAL: Epic puppet fail [21:25:41] PROBLEM - puppet last run on mw1027 is CRITICAL: CRITICAL: Epic puppet fail [21:25:42] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Epic puppet fail [21:25:42] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Epic puppet fail [21:25:52] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Epic puppet fail [21:25:52] PROBLEM - puppet last run on db71 is CRITICAL: CRITICAL: Epic puppet fail [21:26:01] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Epic puppet fail [21:26:02] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Epic puppet fail [21:26:02] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Epic puppet fail [21:26:02] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Epic puppet fail [21:26:03] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Epic puppet fail [21:26:12] PROBLEM - puppet last run on cp4009 is CRITICAL: CRITICAL: Epic puppet fail [21:26:12] PROBLEM - puppet last run on baham is CRITICAL: CRITICAL: Epic puppet fail [21:26:12] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Epic puppet fail [21:26:12] PROBLEM - puppet last run on rbf1001 is CRITICAL: CRITICAL: Epic puppet fail [21:26:12] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Epic puppet fail [21:26:13] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Epic puppet fail [21:26:13] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Epic puppet fail [21:26:13] PROBLEM - puppet last run on lvs3003 is CRITICAL: CRITICAL: Epic puppet fail [21:26:22] PROBLEM - puppet last run on mw1021 is CRITICAL: CRITICAL: Epic puppet fail [21:26:22] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Epic puppet fail [21:26:22] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Epic puppet fail [21:26:22] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Epic puppet fail [21:26:22] PROBLEM - puppet last run on mw1037 is CRITICAL: CRITICAL: Epic puppet fail [21:26:22] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Epic puppet fail [21:26:23] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Epic puppet fail [21:26:23] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Epic puppet fail [21:26:23] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Epic puppet fail [21:26:31] PROBLEM - puppet last run on mw1155 is CRITICAL: CRITICAL: Epic puppet fail [21:26:31] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Epic puppet fail [21:26:31] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Epic puppet fail [21:26:31] PROBLEM - puppet last run on amssq37 is CRITICAL: CRITICAL: Epic puppet fail [21:26:31] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Epic puppet fail [21:26:32] PROBLEM - puppet last run on wtp1008 is CRITICAL: CRITICAL: Epic puppet fail [21:26:32] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Epic puppet fail [21:26:32] PROBLEM - puppet last run on osm-cp1001 is CRITICAL: CRITICAL: Epic puppet fail [21:26:33] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Epic puppet fail [21:26:33] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Epic puppet fail [21:26:34] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Epic puppet fail [21:26:44] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Epic puppet fail [21:26:45] PROBLEM - puppet last run on ms-fe1002 is CRITICAL: CRITICAL: Epic puppet fail [21:26:46] PROBLEM - puppet last run on ms-be1015 is CRITICAL: CRITICAL: Epic puppet fail [21:26:46] PROBLEM - puppet last run on cp1053 is CRITICAL: CRITICAL: Epic puppet fail [21:26:51] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Epic puppet fail [21:26:57] PROBLEM - puppet last run on mexia is CRITICAL: CRITICAL: Epic puppet fail [21:26:57] PROBLEM - puppet last run on mw1131 is CRITICAL: CRITICAL: Epic puppet fail [21:27:01] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Epic puppet fail [21:27:02] PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Epic puppet fail [21:27:02] PROBLEM - puppet last run on mw1113 is CRITICAL: CRITICAL: Epic puppet fail [21:27:11] PROBLEM - puppet last run on lvs4004 is CRITICAL: CRITICAL: Epic puppet fail [21:27:11] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Epic puppet fail [21:27:11] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Epic puppet fail [21:27:11] PROBLEM - puppet last run on cp1045 is CRITICAL: CRITICAL: Epic puppet fail [21:27:11] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Epic puppet fail [21:27:12] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Epic puppet fail [21:27:21] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Epic puppet fail [21:27:23] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Epic puppet fail [21:27:24] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Epic puppet fail [21:27:24] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Epic puppet fail [21:27:24] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Epic puppet fail [21:27:31] PROBLEM - puppet last run on search1003 is CRITICAL: CRITICAL: Epic puppet fail [21:27:32] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Epic puppet fail [21:27:32] PROBLEM - puppet last run on analytics1012 is CRITICAL: CRITICAL: Epic puppet fail [21:27:32] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Epic puppet fail [21:27:32] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Epic puppet fail [21:27:32] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Epic puppet fail [21:27:32] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Epic puppet fail [21:27:32] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Epic puppet fail [21:27:33] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Epic puppet fail [21:27:34] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Epic puppet fail [21:27:34] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Epic puppet fail [21:27:35] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Epic puppet fail [21:27:41] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Epic puppet fail [21:27:41] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Epic puppet fail [21:27:42] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Epic puppet fail [21:27:51] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Epic puppet fail [21:28:01] PROBLEM - puppet last run on virt1005 is CRITICAL: CRITICAL: Epic puppet fail [21:28:12] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Epic puppet fail [21:28:21] PROBLEM - puppet last run on ssl1003 is CRITICAL: CRITICAL: Epic puppet fail [21:28:22] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Epic puppet fail [21:28:22] PROBLEM - puppet last run on cp1066 is CRITICAL: CRITICAL: Epic puppet fail [21:28:22] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Epic puppet fail [21:28:22] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Epic puppet fail [21:28:22] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Epic puppet fail [21:28:23] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Epic puppet fail [21:28:23] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Epic puppet fail [21:28:24] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Epic puppet fail [21:28:25] PROBLEM - puppet last run on lanthanum is CRITICAL: CRITICAL: Epic puppet fail [21:28:25] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Epic puppet fail [21:28:25] PROBLEM - puppet last run on db1049 is CRITICAL: CRITICAL: Epic puppet fail [21:28:26] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Epic puppet fail [21:28:26] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Epic puppet fail [21:28:26] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: Epic puppet fail [21:28:31] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Epic puppet fail [21:28:31] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Epic puppet fail [21:28:31] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Epic puppet fail [21:28:31] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Epic puppet fail [21:28:31] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Epic puppet fail [21:28:31] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Epic puppet fail [21:28:32] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Epic puppet fail [21:28:32] PROBLEM - puppet last run on search1019 is CRITICAL: CRITICAL: Epic puppet fail [21:28:33] PROBLEM - puppet last run on db1005 is CRITICAL: CRITICAL: Epic puppet fail [21:28:33] PROBLEM - puppet last run on elastic1010 is CRITICAL: CRITICAL: Epic puppet fail [21:28:34] PROBLEM - puppet last run on elastic1013 is CRITICAL: CRITICAL: Epic puppet fail [21:28:34] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: Epic puppet fail [21:28:41] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: Epic puppet fail [21:28:41] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Epic puppet fail [21:28:41] PROBLEM - puppet last run on search1008 is CRITICAL: CRITICAL: Epic puppet fail [21:28:42] PROBLEM - puppet last run on virt0 is CRITICAL: CRITICAL: Epic puppet fail [21:28:42] PROBLEM - puppet last run on mw1015 is CRITICAL: CRITICAL: Epic puppet fail [21:29:01] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: Epic puppet fail [21:29:01] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Epic puppet fail [21:29:01] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Epic puppet fail [21:29:01] PROBLEM - puppet last run on zinc is CRITICAL: CRITICAL: Epic puppet fail [21:29:02] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: Epic puppet fail [21:29:02] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: Epic puppet fail [21:29:02] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Epic puppet fail [21:29:11] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Epic puppet fail [21:29:11] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Epic puppet fail [21:29:11] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Epic puppet fail [21:29:11] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Epic puppet fail [21:29:11] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Epic puppet fail [21:29:12] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Epic puppet fail [21:29:12] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Epic puppet fail [21:29:12] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Epic puppet fail [21:29:13] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Epic puppet fail [21:29:13] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Epic puppet fail [21:29:22] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Epic puppet fail [21:29:24] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Epic puppet fail [21:29:24] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: Epic puppet fail [21:29:24] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: Epic puppet fail [21:29:25] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: Epic puppet fail [21:29:31] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Epic puppet fail [21:29:31] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Epic puppet fail [21:29:33] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Epic puppet fail [21:29:33] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Epic puppet fail [21:29:33] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Epic puppet fail [21:29:34] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: Epic puppet fail [21:29:34] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Epic puppet fail [21:29:41] PROBLEM - puppet last run on search1021 is CRITICAL: CRITICAL: Epic puppet fail [21:29:51] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Epic puppet fail [21:29:52] PROBLEM - puppet last run on search1009 is CRITICAL: CRITICAL: Epic puppet fail [21:29:52] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Epic puppet fail [21:29:53] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: Epic puppet fail [21:29:53] PROBLEM - puppet last run on ssl1004 is CRITICAL: CRITICAL: Epic puppet fail [21:30:01] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Epic puppet fail [21:30:01] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: Epic puppet fail [21:30:01] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Epic puppet fail [21:30:02] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: Epic puppet fail [21:30:02] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Epic puppet fail [21:30:11] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Epic puppet fail [21:30:11] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Epic puppet fail [21:30:12] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Epic puppet fail [21:30:12] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Epic puppet fail [21:30:12] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Epic puppet fail [21:30:12] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: Epic puppet fail [21:30:12] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: Epic puppet fail [21:30:23] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Epic puppet fail [21:30:32] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Epic puppet fail [21:30:32] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: Epic puppet fail [21:30:33] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Epic puppet fail [21:30:41] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Epic puppet fail [21:30:41] PROBLEM - puppet last run on search1014 is CRITICAL: CRITICAL: Epic puppet fail [21:30:41] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Epic puppet fail [21:30:51] RECOVERY - ElasticSearch health check on logstash1001 is OK: OK - elasticsearch (production-logstash-eqiad) is running. status: green: timed_out: false: number_of_nodes: 3: number_of_data_nodes: 3: active_primary_shards: 36: active_shards: 103: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [21:30:51] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Epic puppet fail [21:30:53] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: Epic puppet fail [21:30:53] PROBLEM - puppet last run on es10 is CRITICAL: CRITICAL: Epic puppet fail [21:30:53] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Epic puppet fail [21:30:53] PROBLEM - puppet last run on es7 is CRITICAL: CRITICAL: Epic puppet fail [21:30:53] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Epic puppet fail [21:30:53] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Epic puppet fail [21:31:01] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Epic puppet fail [21:31:01] RECOVERY - ElasticSearch health check on logstash1002 is OK: OK - elasticsearch (production-logstash-eqiad) is running. status: green: timed_out: false: number_of_nodes: 3: number_of_data_nodes: 3: active_primary_shards: 36: active_shards: 103: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [21:31:02] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Epic puppet fail [21:31:12] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Epic puppet fail [21:31:12] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Epic puppet fail [21:31:13] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Epic puppet fail [21:31:22] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Epic puppet fail [21:31:22] (03PS1) 10Ori.livneh: Remove salt-minion.override [puppet] - 10https://gerrit.wikimedia.org/r/160301 [21:31:22] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Epic puppet fail [21:31:22] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Epic puppet fail [21:31:31] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Epic puppet fail [21:31:31] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Epic puppet fail [21:31:45] (03CR) 10Ori.livneh: [C: 032 V: 032] Remove salt-minion.override [puppet] - 10https://gerrit.wikimedia.org/r/160301 (owner: 10Ori.livneh) [21:31:51] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Epic puppet fail [21:32:11] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Epic puppet fail [21:36:51] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Epic puppet fail [21:42:52] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [21:43:22] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Epic puppet fail [21:46:11] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [21:51:31] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [21:51:31] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [21:51:31] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [21:51:31] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [21:51:31] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [21:51:31] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [21:51:32] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [21:51:32] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [21:51:33] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [21:51:33] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [21:51:33] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [21:51:34] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [21:51:34] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [21:51:41] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:51:42] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:51:42] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:51:42] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:51:42] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:51:42] RECOVERY - puppet last run on analytics1038 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:51:43] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [21:51:43] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [21:51:43] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:51:44] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:51:44] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:51:44] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:51:45] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:51:45] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:51:46] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:51:46] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:51:47] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:51:48] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:51:48] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:51:49] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:51:49] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [21:51:49] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:51:50] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [21:51:50] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [21:51:51] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [21:51:52] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:51:52] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:51:52] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:51:54] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [21:51:54] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [21:51:54] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [21:51:54] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [21:51:55] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [21:51:55] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [21:51:56] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [21:51:57] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:51:57] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [21:51:57] RECOVERY - puppet last run on analytics1041 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [21:51:58] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:51:58] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:52:01] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [21:52:01] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:52:01] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [21:52:01] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:52:02] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:52:02] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [21:53:21] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [21:53:22] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [21:53:22] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:53:22] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [21:53:22] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [21:53:32] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [21:53:33] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [21:53:41] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [21:53:41] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:53:42] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [21:53:42] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [21:53:53] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [21:54:06] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [21:54:06] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [21:54:06] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [21:54:07] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [21:54:07] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [21:54:14] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:54:15] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [21:54:15] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [21:54:15] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [21:54:15] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [21:54:15] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [21:54:52] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [22:00:21] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:00:22] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:00:46] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [22:00:49] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:00:51] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:01:01] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:01:02] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:01:02] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:01:11] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [22:01:12] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:01:21] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:01:21] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:01:31] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:01:31] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [22:01:31] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:01:32] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:01:32] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:01:32] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:01:41] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:01:41] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:01:41] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:01:42] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [22:01:42] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:01:42] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:02:02] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:02:02] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [22:02:03] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:02:03] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:02:03] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:02:03] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:02:03] RECOVERY - puppet last run on db1017 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:02:03] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:02:04] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:02:11] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [22:02:12] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:02:21] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [22:02:21] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:02:21] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:02:22] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [22:02:22] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:02:22] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [22:02:33] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:02:33] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:02:42] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:02:42] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:02:43] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [22:02:43] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:02:54] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:02:54] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:03:02] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:03:02] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:03:02] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:03:02] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:03:02] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:03:11] RECOVERY - puppet last run on db73 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:03:11] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:03:11] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:03:11] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:03:12] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:03:12] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:03:21] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:03:21] RECOVERY - puppet last run on analytics1040 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:03:22] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:03:22] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:03:22] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:03:22] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:03:22] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:03:33] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:03:33] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:03:41] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:03:41] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:03:41] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:03:41] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:03:42] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:03:42] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:03:42] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:03:42] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:03:43] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [22:03:43] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:03:55] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:03:55] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [22:03:55] RECOVERY - puppet last run on gold is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:04:01] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:04:02] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [22:04:02] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:04:15] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:04:16] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:04:16] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:04:16] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:04:16] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:04:16] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [22:04:24] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:04:25] RECOVERY - puppet last run on db74 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:04:25] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [22:04:25] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [22:04:37] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:04:37] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:04:37] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:04:37] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:04:38] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [22:04:41] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:04:41] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [22:04:52] RECOVERY - puppet last run on analytics1035 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:05:01] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [22:05:02] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:05:11] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [22:05:11] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:05:11] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [22:05:12] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [22:05:23] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:05:23] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:05:23] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:05:32] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:05:42] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:05:43] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:05:43] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [22:05:52] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [22:05:53] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [22:06:01] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:06:01] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:06:01] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:06:01] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [22:06:01] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:06:02] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:06:07] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [22:06:07] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:06:11] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [22:06:16] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:06:31] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:06:32] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [22:06:33] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [22:06:33] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [22:06:41] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:06:41] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:06:41] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:06:41] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:06:51] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:06:52] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:06:52] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [22:07:32] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:07:32] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:07:41] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:07:42] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [22:07:42] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:08:02] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:08:11] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:08:12] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:08:32] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:08:32] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [22:08:35] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [22:08:52] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [22:08:53] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:08:54] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:09:02] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [22:09:11] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [22:09:31] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:09:42] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:10:02] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [22:10:21] RECOVERY - puppet last run on db72 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [22:10:22] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [22:10:53] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:11:31] RECOVERY - puppet last run on analytics1032 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:11:32] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:11:42] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:11:42] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:11:43] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:11:51] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:12:02] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [22:12:21] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:12:22] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:12:32] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [22:12:41] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [22:13:52] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:13:57] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:15:01] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:15:41] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:15:42] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:15:42] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:16:55] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:17:02] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:17:11] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:18:01] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:18:02] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:18:12] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [22:18:31] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:18:32] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:19:41] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: Epic puppet fail [22:22:52] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:39:41] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:39:42] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:40:21] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:40:31] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:40:57] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:41:11] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:41:12] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:41:32] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:41:35] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [22:41:35] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:41:45] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:42:01] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:42:11] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [22:42:31] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [22:42:32] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:42:41] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [22:43:02] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [22:43:21] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:43:21] RECOVERY - puppet last run on platinum is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:43:42] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:44:01] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:44:27] RECOVERY - puppet last run on rbf1002 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [22:44:33] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:44:45] RECOVERY - puppet last run on sca1002 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [22:44:45] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:44:45] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:44:46] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:44:47] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [22:45:14] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:45:24] RECOVERY - puppet last run on xenon is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [22:45:24] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:45:45] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:46:15] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:46:25] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:46:49] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:47:04] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:47:05] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:47:15] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:47:15] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [22:47:22] mmm, spam for dinner [22:47:35] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [22:47:46] RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:47:55] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:48:14] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:48:14] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:48:15] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:48:15] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:48:15] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [22:48:37] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:48:45] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [22:48:46] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:48:47] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:48:56] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [22:48:56] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:49:04] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:49:15] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [22:49:24] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [22:49:36] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [22:49:36] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:49:37] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:49:45] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:50:14] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:50:15] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:50:15] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [22:50:15] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [22:50:15] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:50:24] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [22:50:26] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:50:56] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:50:57] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [22:50:57] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:50:57] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:50:57] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:50:58] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:51:04] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:51:15] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:51:16] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:51:16] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:51:26] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [22:51:35] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:51:55] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:52:14] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:52:14] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:52:15] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:52:15] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:52:15] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [22:52:15] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:52:15] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:52:24] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:52:24] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [22:52:24] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:52:54] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:52:54] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:52:56] RECOVERY - puppet last run on ocg1003 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [22:53:14] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [22:53:14] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:53:15] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:53:26] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:53:40] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:53:55] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [22:53:56] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:53:58] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:54:15] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:54:15] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:54:24] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [22:54:26] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [22:54:34] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:54:35] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:54:44] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:54:58] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:55:08] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [22:55:08] RECOVERY - puppet last run on rbf1001 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [22:55:08] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:55:08] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:55:14] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:55:24] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:55:24] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:55:24] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:55:24] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [22:55:25] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:55:35] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:55:57] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:56:15] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:56:24] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:56:45] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:56:55] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:57:04] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [22:57:45] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [22:58:04] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:58:05] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:58:24] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [22:58:25] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:58:47] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [22:58:55] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [22:58:55] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:58:55] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [22:59:04] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [22:59:25] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:59:25] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:59:44] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [22:59:44] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [22:59:55] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:01:04] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:01:06] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [23:01:25] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:01:26] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [23:01:34] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:19:50] (03Abandoned) 10Jeremyb: new path for beta cluster CommonSettings.php [puppet] - 10https://gerrit.wikimedia.org/r/159487 (owner: 10Jeremyb)