[00:01:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.431 seconds [01:22:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 1.089 seconds [02:01:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 5.184 seconds [05:34:10] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [05:36:07] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [05:43:10] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [05:43:10] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [08:42:33] PROBLEM - Puppet freshness on stafford is CRITICAL: Puppet has not run in the last 10 hours [09:28:54] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [11:14:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:18:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 1.362 seconds [11:53:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:33] PROBLEM - check_minfraud_primary on payments1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:33] PROBLEM - check_minfraud_primary on payments3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:33] PROBLEM - check_minfraud_primary on payments2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:33] PROBLEM - check_minfraud_primary on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:57:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 9.068 seconds [12:00:30] RECOVERY - check_minfraud_primary on payments3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 118 bytes in 0.188 second response time [12:00:30] RECOVERY - check_minfraud_primary on payments1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 118 bytes in 0.182 second response time [12:00:30] RECOVERY - check_minfraud_primary on payments2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 118 bytes in 0.183 second response time [12:00:30] RECOVERY - check_minfraud_primary on payments4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 118 bytes in 0.192 second response time [12:33:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:24] PROBLEM - Host virt3 is DOWN: PING CRITICAL - Packet loss = 100% [12:39:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.546 seconds [13:14:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 7.860 seconds [13:54:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 5.920 seconds [14:34:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.242 seconds [15:14:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.474 seconds [15:35:18] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [15:37:15] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [15:44:18] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [15:44:18] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [15:53:36] PROBLEM - Host cp1019 is DOWN: PING CRITICAL - Packet loss = 100% [15:54:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.243 seconds [16:12:30] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 185 seconds [16:14:36] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [16:36:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:42] PROBLEM - Disk space on search1015 is CRITICAL: DISK CRITICAL - free space: /a 3784 MB (3% inode=99%): [16:40:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.571 seconds [17:15:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 1.505 seconds [17:55:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.162 seconds [18:37:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 9.931 seconds [18:44:31] PROBLEM - Puppet freshness on stafford is CRITICAL: Puppet has not run in the last 10 hours [19:17:13] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 8.788 seconds [19:30:34] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [19:56:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.260 seconds [20:38:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 5.273 seconds [21:18:14] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:18:50] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 320 seconds [21:18:59] PROBLEM - MySQL Slave Delay on db1033 is CRITICAL: CRIT replication delay 327 seconds [21:19:53] PROBLEM - MySQL Slave Delay on db1043 is CRITICAL: CRIT replication delay 382 seconds [21:20:20] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 409 seconds [21:24:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.799 seconds [21:59:11] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.031 seconds [22:32:38] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 236 MB (3% inode=61%): /var/lib/ureadahead/debugfs 236 MB (3% inode=61%): [22:38:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:02] RECOVERY - Disk space on srv220 is OK: DISK OK [22:44:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 7.325 seconds [22:45:41] PROBLEM - Disk space on db42 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=83%): /var/lib/ureadahead/debugfs 0 MB (0% inode=83%): [22:45:59] PROBLEM - MySQL disk space on db42 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=83%): /var/lib/ureadahead/debugfs 0 MB (0% inode=83%): [22:52:53] PROBLEM - RAID on grosley is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:54:59] RECOVERY - RAID on grosley is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [23:18:05] PROBLEM - RAID on grosley is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:20:11] RECOVERY - RAID on grosley is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [23:20:11] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:24:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.149 seconds [23:28:35] PROBLEM - RAID on grosley is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:34:44] RECOVERY - RAID on grosley is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [23:50:51] !log virt3 crashed, powercycling it [23:50:54] Logged the message, Master [23:55:08] RECOVERY - Host virt3 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [23:59:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds