[00:27:21] ok i am headed out of eqiad, rob now offsite. [00:27:26] back online shortly [01:11:39] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [01:29:10] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 152 MB (2% inode=60%): /var/lib/ureadahead/debugfs 152 MB (2% inode=60%): [01:33:20] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 110 MB (1% inode=60%): /var/lib/ureadahead/debugfs 110 MB (1% inode=60%): [01:37:10] RECOVERY - Disk space on srv224 is OK: DISK OK [01:39:20] RECOVERY - Disk space on srv220 is OK: DISK OK [01:53:00] RECOVERY - Disk space on srv222 is OK: DISK OK [02:36:41] PROBLEM - MySQL replication status on db1025 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1272s [03:32:52] RECOVERY - MySQL replication status on db1025 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [04:19:57] RECOVERY - Disk space on es1004 is OK: DISK OK [04:21:07] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:44:37] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:10:48] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [06:55:36] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [07:03:16] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [07:16:16] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [09:04:19] PROBLEM - Puppet freshness on db1003 is CRITICAL: Puppet has not run in the last 10 hours [09:56:30] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 451393 MB (3% inode=99%): [10:00:20] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 424095 MB (3% inode=99%): [10:10:23] RECOVERY - MySQL slave status on es1004 is OK: OK: [15:20:11] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [17:04:26] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [17:11:59] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [17:25:49] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [18:58:27] PROBLEM - Apache HTTP on srv282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:47] PROBLEM - Disk space on srv282 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:06:37] PROBLEM - SSH on srv282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:37] PROBLEM - RAID on srv282 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:06:37] PROBLEM - DPKG on srv282 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:15:11] RECOVERY - Disk space on srv282 is OK: DISK OK [19:20:51] RECOVERY - Apache HTTP on srv282 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [19:20:51] RECOVERY - RAID on srv282 is OK: OK: no RAID installed [19:21:01] PROBLEM - Puppet freshness on db1003 is CRITICAL: Puppet has not run in the last 10 hours [19:22:41] RECOVERY - SSH on srv282 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [19:22:41] RECOVERY - DPKG on srv282 is OK: All packages OK [22:02:49] PROBLEM - Disk space on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:06:19] PROBLEM - RAID on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:06:59] PROBLEM - Apache HTTP on srv262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:19] PROBLEM - SSH on srv262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:19] PROBLEM - DPKG on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:08:39] PROBLEM - RAID on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:09:39] PROBLEM - Apache HTTP on srv276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:19] PROBLEM - SSH on srv276 is CRITICAL: Server answer: [22:13:29] PROBLEM - DPKG on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:14:09] PROBLEM - Disk space on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:29:19] RECOVERY - Apache HTTP on srv276 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [22:30:19] RECOVERY - SSH on srv276 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [22:33:29] RECOVERY - Disk space on srv276 is OK: DISK OK [22:33:29] RECOVERY - DPKG on srv276 is OK: All packages OK [22:38:29] RECOVERY - RAID on srv276 is OK: OK: no RAID installed [22:46:20] RECOVERY - Apache HTTP on srv262 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [22:46:20] RECOVERY - RAID on srv262 is OK: OK: no RAID installed [22:50:10] RECOVERY - Disk space on srv262 is OK: DISK OK [22:50:50] RECOVERY - SSH on srv262 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [22:51:50] RECOVERY - DPKG on srv262 is OK: All packages OK [23:35:50] PROBLEM - Apache HTTP on srv262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:50] PROBLEM - RAID on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:39:50] PROBLEM - Disk space on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:40:10] PROBLEM - SSH on srv262 is CRITICAL: Server answer: [23:41:30] PROBLEM - DPKG on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:49:00] PROBLEM - RAID on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:49:30] RECOVERY - Disk space on srv262 is OK: DISK OK [23:49:50] PROBLEM - DPKG on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:50:00] RECOVERY - SSH on srv262 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:53:20] PROBLEM - Disk space on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:56:40] RECOVERY - Apache HTTP on srv262 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [23:56:40] RECOVERY - RAID on srv262 is OK: OK: no RAID installed [23:58:50] PROBLEM - Apache HTTP on srv276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds