[00:27:12] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [01:02:26] PROBLEM - Puppet freshness on search1013 is CRITICAL: Puppet has not run in the last 10 hours [01:02:26] PROBLEM - Puppet freshness on search1023 is CRITICAL: Puppet has not run in the last 10 hours [01:05:44] PROBLEM - Puppet freshness on search1011 is CRITICAL: Puppet has not run in the last 10 hours [01:08:44] PROBLEM - Puppet freshness on search1010 is CRITICAL: Puppet has not run in the last 10 hours [01:09:38] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [01:11:44] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [01:14:44] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [01:14:44] PROBLEM - Puppet freshness on search1008 is CRITICAL: Puppet has not run in the last 10 hours [01:14:44] PROBLEM - Puppet freshness on search1014 is CRITICAL: Puppet has not run in the last 10 hours [01:17:44] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [01:17:44] PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours [01:17:44] PROBLEM - Puppet freshness on search1005 is CRITICAL: Puppet has not run in the last 10 hours [01:17:44] PROBLEM - Puppet freshness on search1009 is CRITICAL: Puppet has not run in the last 10 hours [01:17:44] PROBLEM - Puppet freshness on search1017 is CRITICAL: Puppet has not run in the last 10 hours [01:17:45] PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours [01:19:41] PROBLEM - Puppet freshness on search1012 is CRITICAL: Puppet has not run in the last 10 hours [01:21:38] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [01:21:38] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [01:21:38] PROBLEM - Puppet freshness on search1020 is CRITICAL: Puppet has not run in the last 10 hours [01:23:44] PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours [01:23:44] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [01:25:41] PROBLEM - Puppet freshness on search1015 is CRITICAL: Puppet has not run in the last 10 hours [01:25:41] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [04:03:20] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Puppet has not run in the last 10 hours [04:17:07] PROBLEM - Router interfaces on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, interfaces up: 73, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-1/1/0: down - Core: cr2-eqiad:xe-5/2/1 (FPL/Level3, CV71028) [10Gbps wave]BR [04:52:58] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 4324 MB (3% inode=99%): [04:53:07] PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 3343 MB (2% inode=99%): [05:06:36] PROBLEM - MySQL Slave Delay on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:06:36] PROBLEM - DPKG on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:07:12] PROBLEM - Disk space on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:08:06] PROBLEM - RAID on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:08:24] PROBLEM - SSH on db25 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:24] PROBLEM - MySQL Replication Heartbeat on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:08:42] PROBLEM - mysqld processes on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:09:27] PROBLEM - MySQL Slave Running on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:09:27] PROBLEM - MySQL disk space on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:09:27] PROBLEM - Full LVS Snapshot on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:09:45] PROBLEM - MySQL Idle Transactions on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:10:12] PROBLEM - MySQL Recent Restart on db25 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:34:48] PROBLEM - NTP on db25 is CRITICAL: NTP CRITICAL: No response from NTP server [06:08:17] PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 3728 MB (3% inode=99%): [06:08:17] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 3714 MB (3% inode=99%): [06:18:47] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 3709 MB (3% inode=99%): [06:22:05] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [06:33:02] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [06:33:02] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [06:47:06] PROBLEM - Squid on brewster is CRITICAL: Connection refused [06:47:33] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [06:47:33] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [07:31:29] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [08:43:15] RECOVERY - NTP on db25 is OK: NTP OK: Offset -0.00320315361 secs [08:43:33] RECOVERY - RAID on db25 is OK: OK: 1 logical device(s) checked [08:43:51] RECOVERY - SSH on db25 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [08:44:18] RECOVERY - mysqld processes on db25 is OK: PROCS OK: 1 process with command name mysqld [08:44:27] RECOVERY - DPKG on db25 is OK: All packages OK [08:44:45] RECOVERY - Disk space on db25 is OK: DISK OK [08:44:54] RECOVERY - Full LVS Snapshot on db25 is OK: OK no full LVM snapshot volumes [08:44:54] RECOVERY - MySQL disk space on db25 is OK: DISK OK [08:45:30] RECOVERY - MySQL Idle Transactions on db25 is OK: OK longest blocking idle transaction sleeps for 0 seconds [08:45:57] RECOVERY - MySQL Slave Running on db25 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [08:46:51] RECOVERY - MySQL Recent Restart on db25 is OK: OK 4970415 seconds since restart [09:02:45] RECOVERY - MySQL Replication Heartbeat on db25 is OK: OK replication delay 0 seconds [09:02:45] RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 75, down: 0, dormant: 0, excluded: 0, unused: 0 [09:03:12] RECOVERY - MySQL Slave Delay on db25 is OK: OK replication delay 0 seconds [09:29:26] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [10:05:08] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:07:15] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [10:29:10] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [11:04:07] PROBLEM - Puppet freshness on search1023 is CRITICAL: Puppet has not run in the last 10 hours [11:04:07] PROBLEM - Puppet freshness on search1013 is CRITICAL: Puppet has not run in the last 10 hours [11:07:07] PROBLEM - Puppet freshness on search1011 is CRITICAL: Puppet has not run in the last 10 hours [11:10:07] PROBLEM - Puppet freshness on search1010 is CRITICAL: Puppet has not run in the last 10 hours [11:11:10] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [11:12:13] PROBLEM - HTTP on formey is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:07] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [11:16:07] PROBLEM - Puppet freshness on search1008 is CRITICAL: Puppet has not run in the last 10 hours [11:16:07] PROBLEM - Puppet freshness on search1014 is CRITICAL: Puppet has not run in the last 10 hours [11:16:07] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [11:19:07] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [11:19:07] PROBLEM - Puppet freshness on search1009 is CRITICAL: Puppet has not run in the last 10 hours [11:19:07] PROBLEM - Puppet freshness on search1005 is CRITICAL: Puppet has not run in the last 10 hours [11:19:07] PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours [11:19:07] PROBLEM - Puppet freshness on search1017 is CRITICAL: Puppet has not run in the last 10 hours [11:19:08] PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours [11:21:30] PROBLEM - Puppet freshness on search1012 is CRITICAL: Puppet has not run in the last 10 hours [11:23:18] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [11:23:18] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [11:23:18] PROBLEM - Puppet freshness on search1020 is CRITICAL: Puppet has not run in the last 10 hours [11:25:33] PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours [11:25:33] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [11:27:39] PROBLEM - Puppet freshness on search1015 is CRITICAL: Puppet has not run in the last 10 hours [11:27:39] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [11:34:51] PROBLEM - HTTPS on formey is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:54] RECOVERY - HTTPS on formey is OK: OK - Certificate will expire on 08/22/2015 22:23. [11:49:15] RECOVERY - HTTP on formey is OK: HTTP OK HTTP/1.1 200 OK - 3596 bytes in 0.789 seconds [12:46:53] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 218 MB (3% inode=63%): [12:51:14] RECOVERY - Disk space on srv221 is OK: DISK OK [14:05:22] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Puppet has not run in the last 10 hours [14:50:22] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [15:12:22] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [15:12:44] * schoolcraftT slaps Krinkle with a large smelly trout [16:18:30] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [16:23:45] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [16:34:33] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [16:34:33] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [16:49:33] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [16:49:33] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [17:32:36] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [19:30:32] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [19:50:28] * schoolcraftT slaps Laaknor a few times [20:30:37] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [21:05:43] PROBLEM - Puppet freshness on search1023 is CRITICAL: Puppet has not run in the last 10 hours [21:05:43] PROBLEM - Puppet freshness on search1013 is CRITICAL: Puppet has not run in the last 10 hours [21:08:43] PROBLEM - Puppet freshness on search1011 is CRITICAL: Puppet has not run in the last 10 hours [21:11:43] PROBLEM - Puppet freshness on search1010 is CRITICAL: Puppet has not run in the last 10 hours [21:12:46] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [21:14:43] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [21:17:43] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [21:17:43] PROBLEM - Puppet freshness on search1008 is CRITICAL: Puppet has not run in the last 10 hours [21:17:43] PROBLEM - Puppet freshness on search1014 is CRITICAL: Puppet has not run in the last 10 hours [21:20:43] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [21:20:43] PROBLEM - Puppet freshness on search1009 is CRITICAL: Puppet has not run in the last 10 hours [21:20:43] PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours [21:20:43] PROBLEM - Puppet freshness on search1017 is CRITICAL: Puppet has not run in the last 10 hours [21:20:43] PROBLEM - Puppet freshness on search1005 is CRITICAL: Puppet has not run in the last 10 hours [21:20:44] PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours [21:22:40] PROBLEM - Puppet freshness on search1012 is CRITICAL: Puppet has not run in the last 10 hours [21:24:37] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [21:24:37] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [21:24:37] PROBLEM - Puppet freshness on search1020 is CRITICAL: Puppet has not run in the last 10 hours [21:26:43] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [21:26:43] PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours [21:29:43] PROBLEM - Puppet freshness on search1015 is CRITICAL: Puppet has not run in the last 10 hours [21:29:43] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [21:33:10] . [21:49:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.739 seconds [22:26:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.872 seconds [23:08:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:12:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.581 seconds [23:48:10] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 8.577 seconds