[00:01:37] 2013/07/24 23:54 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [00:04:39] 2013/07/25 00:03 OK yucca SMTP SMTP OK - 3.007 sec. response time [00:05:37] 2013/07/25 00:04 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [00:14:38] 2013/07/25 00:07 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [00:15:43] 2013/07/25 00:08 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [00:19:39] 2013/07/25 00:18 OK yucca SMTP SMTP OK - 3.007 sec. response time [00:20:39] 2013/07/25 00:19 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [00:30:40] 2013/07/25 00:23 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [00:30:40] 2013/07/25 00:23 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [00:32:40] 2013/07/25 00:31 OK yucca SMTP SMTP OK - 3.005 sec. response time [00:33:40] 2013/07/25 00:32 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [00:39:41] 2013/07/25 00:32 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [00:40:42] 2013/07/25 00:33 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [00:45:41] 2013/07/25 00:44 OK yucca SMTP SMTP OK - 3.005 sec. response time [00:46:41] 2013/07/25 00:45 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [00:53:41] 2013/07/25 00:46 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [00:54:41] 2013/07/25 00:53 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [01:10:44] 2013/07/25 01:03 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [01:11:44] 2013/07/25 01:04 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [01:32:44] 2013/07/25 01:31 OK yucca SMTP SMTP OK - 3.001 sec. response time [01:33:44] 2013/07/25 01:32 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [01:39:45] 2013/07/25 01:32 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [01:40:45] 2013/07/25 01:39 OK yucca SMTP SMTP OK - 3.004 sec. response time [02:10:47] 2013/07/25 02:03 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [02:11:47] 2013/07/25 02:04 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [02:22:48] 2013/07/25 02:21 OK yucca SMTP SMTP OK - 3.007 sec. response time [02:24:48] 2013/07/25 02:24 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [02:47:49] 2013/07/25 02:40 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [02:48:49] 2013/07/25 02:41 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [02:51:49] 2013/07/25 02:50 OK yucca SMTP SMTP OK - 3.004 sec. response time [02:51:49] 2013/07/25 02:51 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [03:13:50] 2013/07/25 03:06 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [03:19:50] 2013/07/25 03:18 OK yucca SMTP SMTP OK - 3.004 sec. response time [03:20:51] 2013/07/25 03:13 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [03:21:50] 2013/07/25 03:21 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [04:38:55] 2013/07/25 04:31 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [04:39:54] 2013/07/25 04:32 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [04:41:53] 2013/07/25 04:40 OK yucca SMTP SMTP OK - 3.003 sec. response time [04:41:53] 2013/07/25 04:41 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [04:48:53] 2013/07/25 04:41 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [04:49:54] 2013/07/25 04:48 OK yucca SMTP SMTP OK - 3.007 sec. response time [05:03:54] 2013/07/25 04:56 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [05:04:54] 2013/07/25 04:57 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [05:06:54] 2013/07/25 05:05 OK yucca SMTP SMTP OK - 3.005 sec. response time [05:13:55] 2013/07/25 05:06 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [05:16:01] 2013/07/25 05:14 OK yucca SMTP SMTP OK - 3.007 sec. response time [05:16:01] 2013/07/25 05:15 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [05:30:56] 2013/07/25 05:24 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [05:31:56] 2013/07/25 05:24 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [06:03:56] 2013/07/25 06:02 OK yucca SMTP SMTP OK - 3.008 sec. response time [06:03:56] 2013/07/25 06:03 OK z-dat-s2-b SMTP SMTP OK - 0.107 sec. response time [06:10:56] 2013/07/25 06:03 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [06:10:57] 2013/07/25 06:04 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [06:22:57] 2013/07/25 06:21 OK yucca SMTP SMTP OK - 3.005 sec. response time [06:25:56] 2013/07/25 06:25 OK z-dat-s2-b SMTP SMTP OK - 0.095 sec. response time [06:38:56] 2013/07/25 06:31 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [06:38:56] 2013/07/25 06:32 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [06:44:57] 2013/07/25 06:43 OK yucca SMTP SMTP OK - 3.009 sec. response time [06:44:57] 2013/07/25 06:44 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [06:57:57] 2013/07/25 06:50 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [07:02:59] 2013/07/25 06:56 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [07:13:59] 2013/07/25 07:12 OK yucca SMTP SMTP OK - 3.003 sec. response time [07:13:59] 2013/07/25 07:13 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [07:20:58] 2013/07/25 07:14 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [07:21:58] 2013/07/25 07:21 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [07:26:58] 2013/07/25 07:19 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [07:28:58] 2013/07/25 07:22 CRIT ortelius Load avg. CRITICAL - load average: 8.71, 54.69, 35.86 [07:28:58] 2013/07/25 07:22 CRIT wolfsbane Load avg. CRITICAL - load average: 23.26, 77.62, 46.47 [07:28:59] 2013/07/25 07:27 OK yucca SMTP SMTP OK - 3.003 sec. response time [07:35:59] 2013/07/25 07:28 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [07:35:59] 2013/07/25 07:29 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [07:39:00] 2013/07/25 07:38 WARN ortelius Load avg. WARNING - load average: 2.05, 9.52, 19.65 [07:44:00] 2013/07/25 07:43 OK ortelius Load avg. OK - load average: 3.08, 5.25, 14.87 [07:44:00] 2013/07/25 07:43 WARN wolfsbane Load avg. WARNING - load average: 3.36, 6.91, 19.23 [07:47:00] 2013/07/25 07:45 OK yucca SMTP SMTP OK - 3.002 sec. response time [07:47:00] 2013/07/25 07:46 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [07:49:00] 2013/07/25 07:48 OK wolfsbane Load avg. OK - load average: 3.30, 4.66, 14.77 [07:55:00] 2013/07/25 07:53 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [08:00:00] 2013/07/25 07:59 OK yarrow aliasd TCP OK - 0.003 second response time on port 984 [500 Not found.] [08:00:00] 2013/07/25 07:53 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [08:00:00] 2013/07/25 07:53 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [08:00:00] 2013/07/25 07:53 CRIT z-dat-s2-b MySQL slave Can't connect to MySQL server on 'z-dat-s2-b' (111) [08:06:01] 2013/07/25 08:05 OK z-dat-s2-b /sql DISK OK - free space: /sql 151459 MB (15% inode=99%): [08:06:01] 2013/07/25 08:05 OK z-dat-s2-b MySQL Uptime: 36803 Threads: 18 Questions: 10348128 Slow queries: 420 Opens: 92098 Flush tables: 1 Open tables: 256 Queries per second avg: 281.176 [08:06:01] 2013/07/25 08:05 OK z-dat-s2-b MySQL slave Uptime: 36811 Threads: 18 Questions: 10348207 Slow queries: 420 Opens: 92098 Flush tables: 1 Open tables: 256 Queries per second avg: 281.117 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [08:13:00] 2013/07/25 08:05 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [08:16:00] 2013/07/25 08:08 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [08:16:00] 2013/07/25 08:09 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [08:23:00] 2013/07/25 08:21 OK yucca SMTP SMTP OK - 3.008 sec. response time [08:23:00] 2013/07/25 08:22 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [08:30:01] 2013/07/25 08:22 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [08:30:01] 2013/07/25 08:23 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [08:43:03] 2013/07/25 08:42 OK z-dat-s2-b SMTP SMTP OK - 0.002 sec. response time [08:45:03] 2013/07/25 08:43 OK yucca SMTP SMTP OK - 3.003 sec. response time [08:54:06] 2013/07/25 08:46 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [08:55:05] 2013/07/25 08:54 OK yucca SMTP SMTP OK - 3.007 sec. response time [08:56:05] 2013/07/25 08:55 CRIT yucca / Timeout while attempting connection [08:56:05] 2013/07/25 08:55 CRIT yucca /tmp Connection refused or timed out [08:56:06] 2013/07/25 08:55 CRIT yucca APT Connection refused or timed out [08:56:06] 2013/07/25 08:55 CRIT yucca Environment IPMI Connection refused or timed out [08:56:06] 2013/07/25 08:55 CRIT yucca NTP CRITICAL - Socket timeout after 10 seconds [08:56:06] 2013/07/25 08:55 CRIT yucca PING CRITICAL - Host Unreachable (yucca) [08:56:06] 2013/07/25 08:55 CRIT yucca SMTP No route to host [08:56:07] 2013/07/25 08:55 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [08:57:05] 2013/07/25 08:56 CRIT yucca Load avg. Connection refused or timed out [08:57:06] 2013/07/25 08:56 CRIT yucca SSH No route to host [08:58:05] 2013/07/25 08:57 CRIT yucca CRITICAL - Host Unreachable (yucca) [09:01:05] 2013/07/25 08:55 CRIT z-dat-s2-b MySQL slave Can't connect to MySQL server on 'z-dat-s2-b' (111) [09:01:05] 2013/07/25 08:54 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [09:02:05] 2013/07/25 08:55 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [09:03:05] 2013/07/25 09:02 OK yucca PING OK - Packet loss = 0%, RTA = 0.32 ms [09:03:05] 2013/07/25 09:02 OK yucca / DISK OK - free space: / 1401 MB (78% inode=89%): [09:03:05] 2013/07/25 09:02 OK yucca /tmp DISK OK - free space: /tmp 4316 MB (96% inode=99%): [09:03:05] 2013/07/25 09:02 OK yucca APT APT OK: 0 packages available for upgrade (0 critical updates). [09:03:05] 2013/07/25 09:02 OK yucca Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [09:03:06] 2013/07/25 09:02 OK yucca Load avg. OK - load average: 0.12, 0.06, 0.06 [09:03:06] 2013/07/25 09:02 OK yucca PING PING OK - Packet loss = 0%, RTA = 0.11 ms [09:03:07] 2013/07/25 09:02 OK yucca SMTP SMTP OK - 3.024 sec. response time [09:03:07] 2013/07/25 08:55 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [09:03:08] 2013/07/25 08:55 CRIT z-dat-s2-b SSH CRITICAL - Socket timeout after 10 seconds [09:04:05] 2013/07/25 09:03 CRIT yucca Load avg. Timeout while attempting connection [09:04:05] 2013/07/25 09:03 CRIT yucca PING PING CRITICAL - Packet loss = 100% [09:05:05] 2013/07/25 09:04 CRIT yucca / Timeout while attempting connection [09:05:05] 2013/07/25 09:03 CRIT yucca /tmp Timeout while attempting connection [09:05:05] 2013/07/25 09:03 CRIT yucca APT Timeout while attempting connection [09:05:05] 2013/07/25 09:03 CRIT yucca Environment IPMI Timeout while attempting connection [09:05:05] 2013/07/25 09:03 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [09:05:06] 2013/07/25 09:04 CRIT z-dat-s2-b / Timeout while attempting connection [09:05:06] 2013/07/25 09:03 CRIT z-dat-s2-b /tmp Timeout while attempting connection [09:05:07] 2013/07/25 09:03 CRIT z-dat-s2-b APT Timeout while attempting connection [09:05:07] 2013/07/25 09:04 CRIT z-dat-s2-b Load avg. Timeout while attempting connection [09:05:08] 2013/07/25 09:04 CRIT z-dat-s2-b PING PING CRITICAL - Packet loss = 100% [09:06:05] 2013/07/25 09:05 OK z-dat-s2-b / DISK OK - free space: / 1401 MB (78% inode=89%): [09:06:05] 2013/07/25 09:05 OK z-dat-s2-b /tmp DISK OK - free space: /tmp 4316 MB (96% inode=99%): [09:06:05] 2013/07/25 09:05 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [09:06:05] 2013/07/25 09:05 OK z-dat-s2-b Load avg. OK - load average: 0.01, 0.04, 0.05 [09:06:06] 2013/07/25 09:05 OK z-dat-s2-b PING PING OK - Packet loss = 0%, RTA = 0.31 ms [09:06:06] 2013/07/25 09:05 OK z-dat-s2-b SMTP SMTP OK - 0.031 sec. response time [09:06:06] 2013/07/25 09:05 OK z-dat-s2-b SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [09:07:05] 2013/07/25 09:06 CRIT yucca CRITICAL - Host Unreachable (yucca) [09:09:05] 2013/07/25 08:55 CRIT yucca NTP CRITICAL - Socket timeout after 10 seconds [09:09:06] 2013/07/25 08:56 CRIT yucca SSH No route to host [09:10:05] 2013/07/25 09:09 OK z-dat-s2-b /sql DISK OK - free space: /sql 152275 MB (15% inode=99%): [09:10:06] 2013/07/25 09:09 OK z-dat-s2-b MySQL Uptime: 20 Threads: 6 Questions: 1338 Slow queries: 0 Opens: 1874 Flush tables: 1 Open tables: 42 Queries per second avg: 66.900 [09:10:06] 2013/07/25 09:09 OK z-dat-s2-b MySQL slave Uptime: 28 Threads: 6 Questions: 1468 Slow queries: 0 Opens: 2643 Flush tables: 1 Open tables: 76 Queries per second avg: 52.428 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 877 [09:17:05] 2013/07/25 09:10 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [09:17:05] 2013/07/25 09:10 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [09:17:05] 2013/07/25 09:10 CRIT z-dat-s2-b MySQL slave Can't connect to MySQL server on 'z-dat-s2-b' (111) [10:11:09] 2013/07/25 10:10 OK z-dat-s2-b /sql DISK OK - free space: /sql 152209 MB (15% inode=99%): [10:11:09] 2013/07/25 10:10 OK z-dat-s2-b MySQL Uptime: 3680 Threads: 12 Questions: 72080 Slow queries: 2 Opens: 24691 Flush tables: 1 Open tables: 256 Queries per second avg: 19.586 [10:11:09] 2013/07/25 10:10 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3508 [10:17:10] 2013/07/25 10:10 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [10:32:11] 2013/07/25 10:31 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3654 [10:53:12] 2013/07/25 10:52 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3570 [11:11:12] 2013/07/25 11:10 OK z-dat-s2-b MySQL slave Uptime: 7288 Threads: 17 Questions: 2367280 Slow queries: 66 Opens: 76049 Flush tables: 1 Open tables: 256 Queries per second avg: 324.818 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1369 [11:16:13] 2013/07/25 11:15 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [11:23:13] 2013/07/25 11:16 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [11:23:13] 2013/07/25 11:16 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [11:23:13] 2013/07/25 11:16 CRIT z-dat-s2-b MySQL slave Can't connect to MySQL server on 'z-dat-s2-b' (111) [12:17:14] 2013/07/25 12:16 OK z-dat-s2-b /sql DISK OK - free space: /sql 153098 MB (16% inode=99%): [12:17:14] 2013/07/25 12:16 OK z-dat-s2-b MySQL Uptime: 11240 Threads: 31 Questions: 2540513 Slow queries: 68 Opens: 81697 Flush tables: 1 Open tables: 256 Queries per second avg: 226.24 [12:22:14] 2013/07/25 12:15 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [12:24:14] 2013/07/25 12:23 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [12:30:14] 2013/07/25 12:23 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [12:30:14] 2013/07/25 12:23 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [12:51:15] 2013/07/25 12:50 OK z-dat-s2-b /sql DISK OK - free space: /sql 153098 MB (16% inode=99%): [12:51:16] 2013/07/25 12:50 OK z-dat-s2-b MySQL Uptime: 13280 Threads: 38 Questions: 2569932 Slow queries: 73 Opens: 81887 Flush tables: 1 Open tables: 256 Queries per second avg: 193.518 [12:58:15] 2013/07/25 12:51 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [12:58:15] 2013/07/25 12:51 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [13:12:15] 2013/07/25 13:11 OK z-dat-s2-b /sql DISK OK - free space: /sql 153098 MB (16% inode=99%): [13:12:15] 2013/07/25 13:11 OK z-dat-s2-b MySQL Uptime: 14540 Threads: 41 Questions: 2574096 Slow queries: 73 Opens: 81896 Flush tables: 1 Open tables: 256 Queries per second avg: 177.35 [13:18:16] 2013/07/25 13:11 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [13:24:16] 2013/07/25 13:23 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [13:32:17] 2013/07/25 13:25 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [13:41:18] 2013/07/25 13:40 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [13:48:19] 2013/07/25 13:41 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [13:48:19] 2013/07/25 13:41 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [13:56:20] 2013/07/25 13:55 OK z-dat-s2-b /sql DISK OK - free space: /sql 153098 MB (16% inode=99%): [13:56:20] 2013/07/25 13:55 OK z-dat-s2-b MySQL Uptime: 17180 Threads: 40 Questions: 2725512 Slow queries: 86 Opens: 82076 Flush tables: 1 Open tables: 256 Queries per second avg: 158.644 [13:59:20] 2013/07/25 13:52 CRIT nightshade aliasd Connection refused [14:01:21] 2013/07/25 13:54 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [14:21:21] 2013/07/25 14:14 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2123 [14:21:21] 2013/07/25 14:14 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2114.000000 [14:21:21] 2013/07/25 14:20 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [14:28:21] 2013/07/25 14:21 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [14:28:22] 2013/07/25 14:21 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [14:30:21] 2013/07/25 14:29 OK z-dat-s2-b /sql DISK OK - free space: /sql 153098 MB (16% inode=99%): [14:30:21] 2013/07/25 14:29 OK z-dat-s2-b MySQL Uptime: 19220 Threads: 40 Questions: 2849476 Slow queries: 93 Opens: 82315 Flush tables: 1 Open tables: 256 Queries per second avg: 148.255 [14:36:21] 2013/07/25 14:29 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [14:41:22] 2013/07/25 14:40 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [14:48:23] 2013/07/25 14:41 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [14:48:23] 2013/07/25 14:41 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [14:51:25] 2013/07/25 14:50 OK z-dat-s2-b /sql DISK OK - free space: /sql 153098 MB (16% inode=99%): [14:51:25] 2013/07/25 14:50 OK z-dat-s2-b MySQL Uptime: 20480 Threads: 46 Questions: 2922992 Slow queries: 98 Opens: 82539 Flush tables: 1 Open tables: 256 Queries per second avg: 142.724 [14:52:25] 2013/07/25 14:51 CRIT daphne MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3619 [14:52:25] 2013/07/25 14:51 CRIT daphne s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3608.000000 [15:03:24] 2013/07/25 14:56 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [15:11:24] 2013/07/25 15:10 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [15:18:25] 2013/07/25 15:11 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [15:18:25] 2013/07/25 15:11 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [15:31:25] 2013/07/25 15:30 OK z-dat-s2-b /sql DISK OK - free space: /sql 153098 MB (16% inode=99%): [15:31:25] 2013/07/25 15:30 OK z-dat-s2-b MySQL Uptime: 22880 Threads: 54 Questions: 3033333 Slow queries: 106 Opens: 82701 Flush tables: 1 Open tables: 256 Queries per second avg: 132.575 [15:37:25] 2013/07/25 15:30 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [17:46:34] 2013/07/25 17:45 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [17:53:34] 2013/07/25 17:46 CRIT z-dat-s2-b /sql NRPE: Command 'check_sql' not defined [17:53:35] 2013/07/25 17:46 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [18:33:36] 2013/07/25 18:30 WARN wolfsbane Load avg. WARNING - load average: 5.93, 15.63, 11.71 [18:34:36] 2013/07/25 18:34 OK wolfsbane Load avg. OK - load average: 3.99, 13.33, 11.14 [19:17:40] 2013/07/25 19:17 OK yucca / DISK OK - free space: / 1401 MB (78% inode=89%): [19:19:40] 2013/07/25 19:18 OK yucca NTP NTP OK: Offset 0.002658 secs [19:22:54] 2013/07/25 19:22 CRIT z-dat-s2-b Load avg. Connection refused or timed out [19:23:40] 2013/07/25 19:22 CRIT yucca /tmp Timeout while attempting connection [19:23:40] 2013/07/25 19:22 CRIT yucca APT Timeout while attempting connection [19:23:40] 2013/07/25 19:22 CRIT yucca Environment IPMI Timeout while attempting connection [19:23:40] 2013/07/25 19:22 CRIT yucca Load avg. Timeout while attempting connection [19:23:40] 2013/07/25 19:22 CRIT yucca NTP CRITICAL - Socket timeout after 10 seconds [19:23:40] 2013/07/25 19:22 CRIT yucca PING PING CRITICAL - Packet loss = 100% [19:23:41] 2013/07/25 19:22 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [19:23:41] 2013/07/25 19:23 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [19:23:42] 2013/07/25 19:22 CRIT z-dat-s2-b / Timeout while attempting connection [19:23:42] 2013/07/25 19:22 CRIT z-dat-s2-b /tmp Timeout while attempting connection [19:23:43] 2013/07/25 19:22 CRIT z-dat-s2-b APT Timeout while attempting connection [19:23:43] 2013/07/25 19:22 CRIT z-dat-s2-b PING PING CRITICAL - Packet loss = 100% [19:23:44] 2013/07/25 19:22 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [19:23:44] 2013/07/25 19:22 CRIT z-dat-s2-b SSH CRITICAL - Socket timeout after 10 seconds [19:24:44] 2013/07/25 19:23 CRIT yucca / Timeout while attempting connection [19:26:40] 2013/07/25 19:25 OK z-dat-s2-b /tmp DISK OK - free space: /tmp 4316 MB (96% inode=99%): [19:26:40] 2013/07/25 19:25 OK z-dat-s2-b APT APT OK: 0 packages available for upgrade (0 critical updates). [19:26:40] 2013/07/25 19:26 OK z-dat-s2-b Load avg. OK - load average: 0.00, 0.01, 0.05 [19:26:40] 2013/07/25 19:25 WARN z-dat-s2-b PING PING WARNING - Packet loss = 50%, RTA = 0.18 ms [19:26:40] 2013/07/25 19:25 OK z-dat-s2-b SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [19:27:40] 2013/07/25 19:26 CRIT yucca CRITICAL - Host Unreachable (yucca) [19:27:40] 2013/07/25 19:26 OK z-dat-s2-b / DISK OK - free space: / 1401 MB (78% inode=89%): [19:27:40] 2013/07/25 19:26 OK z-dat-s2-b PING PING OK - Packet loss = 0%, RTA = 0.18 ms [19:27:40] 2013/07/25 19:26 OK z-dat-s2-b SMTP SMTP OK - 0.025 sec. response time [19:47:41] 2013/07/25 19:47 OK yucca PING OK - Packet loss = 0%, RTA = 0.39 ms [19:47:41] 2013/07/25 19:47 OK yucca / DISK OK - free space: / 1401 MB (78% inode=89%): [19:47:41] 2013/07/25 19:46 OK yucca Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [19:47:41] 2013/07/25 19:46 OK yucca SMTP SMTP OK - 3.022 sec. response time [19:47:41] 2013/07/25 19:47 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [19:48:41] 2013/07/25 19:47 OK yucca /tmp DISK OK - free space: /tmp 4316 MB (96% inode=99%): [19:48:41] 2013/07/25 19:47 OK yucca APT APT OK: 0 packages available for upgrade (0 critical updates). [19:48:41] 2013/07/25 19:47 OK yucca Load avg. OK - load average: 0.18, 0.12, 0.08 [19:48:41] 2013/07/25 19:47 OK yucca PING PING OK - Packet loss = 0%, RTA = 0.16 ms [19:49:41] 2013/07/25 19:48 CRIT yucca / Timeout while attempting connection [19:49:41] 2013/07/25 19:48 CRIT yucca /tmp Connection refused or timed out [19:49:41] 2013/07/25 19:48 CRIT yucca APT Connection refused or timed out [19:49:41] 2013/07/25 19:48 CRIT yucca Environment IPMI Connection refused or timed out [19:49:41] 2013/07/25 19:48 CRIT yucca Load avg. Timeout while attempting connection [19:49:41] 2013/07/25 19:48 CRIT yucca PING CRITICAL - Host Unreachable (yucca) [19:49:42] 2013/07/25 19:48 CRIT yucca SMTP No route to host [19:49:42] 2013/07/25 19:48 CRIT z-dat-s2-b / Timeout while attempting connection [19:49:43] 2013/07/25 19:48 CRIT z-dat-s2-b APT Connection refused or timed out [19:49:43] 2013/07/25 19:49 CRIT z-dat-s2-b Load avg. Connection refused or timed out [19:49:44] 2013/07/25 19:48 CRIT z-dat-s2-b PING CRITICAL - Host Unreachable (z-dat-s2-b) [19:49:44] 2013/07/25 19:48 CRIT z-dat-s2-b SSH No route to host [19:50:43] 2013/07/25 19:50 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [19:50:43] 2013/07/25 19:49 CRIT z-dat-s2-b /tmp Connection refused or timed out [19:50:43] 2013/07/25 19:49 CRIT z-dat-s2-b SMTP No route to host [19:52:44] 2013/07/25 19:52 CRIT z-dat-s2-b CRITICAL - Host Unreachable (z-dat-s2-b) [19:54:44] 2013/07/25 19:54 CRIT yucca CRITICAL - Host Unreachable (yucca) [19:54:44] 2013/07/25 19:22 CRIT yucca NTP CRITICAL - Socket timeout after 10 seconds [20:00:45] 2013/07/25 19:59 OK yucca PING OK - Packet loss = 0%, RTA = 0.46 ms [20:00:45] 2013/07/25 20:00 OK yucca / DISK OK - free space: / 1401 MB (78% inode=89%): [20:00:45] 2013/07/25 19:59 OK yucca /tmp DISK OK - free space: /tmp 4317 MB (96% inode=99%): [20:00:45] 2013/07/25 19:59 OK yucca APT APT OK: 0 packages available for upgrade (0 critical updates). [20:00:45] 2013/07/25 19:59 OK yucca Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [20:00:46] 2013/07/25 19:59 OK yucca PING PING OK - Packet loss = 0%, RTA = 0.37 ms [20:00:46] 2013/07/25 19:59 OK yucca SMTP SMTP OK - 0.142 sec. response time [20:00:46] 2013/07/25 20:00 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [20:01:45] 2013/07/25 20:00 OK yucca Load avg. OK - load average: 0.40, 0.17, 0.06 [20:01:45] 2013/07/25 20:00 WARN yucca NTP NTP WARNING: Server has the LI_ALARM bit set, Offset 0.000437 secs [20:08:46] 2013/07/25 20:07 CRIT yucca APT Timeout while attempting connection [20:08:46] 2013/07/25 20:07 CRIT yucca Environment IPMI Timeout while attempting connection [20:08:46] 2013/07/25 20:07 CRIT yucca NTP CRITICAL - Socket timeout after 10 seconds [20:08:46] 2013/07/25 20:08 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [20:08:46] 2013/07/25 20:08 OK z-dat-s2-b Load avg. OK - load average: 0.57, 0.13, 0.04 [20:09:45] 2013/07/25 20:08 OK z-dat-s2-b PING OK - Packet loss = 0%, RTA = 0.30 ms [20:09:45] 2013/07/25 20:08 CRIT yucca / Connection refused or timed out [20:09:45] 2013/07/25 20:08 CRIT yucca /tmp Connection refused or timed out [20:09:45] 2013/07/25 20:08 CRIT yucca Load avg. Connection refused or timed out [20:09:45] 2013/07/25 20:08 CRIT yucca PING CRITICAL - Host Unreachable (yucca) [20:09:45] 2013/07/25 20:08 CRIT yucca SMTP No route to host [20:09:45] 2013/07/25 20:08 OK z-dat-s2-b / DISK OK - free space: / 2570 MB (67% inode=89%): [20:09:46] 2013/07/25 20:08 OK z-dat-s2-b /sql DISK OK - free space: /sql 153002 MB (15% inode=99%): [20:09:46] 2013/07/25 20:08 OK z-dat-s2-b /tmp DISK OK - free space: / 2570 MB (67% inode=89%): [20:09:47] 2013/07/25 20:08 OK z-dat-s2-b MySQL Uptime: 20 Threads: 7 Questions: 145 Slow queries: 0 Opens: 2493 Flush tables: 1 Open tables: 36 Queries per second avg: 7.250 [20:09:47] 2013/07/25 20:08 OK z-dat-s2-b PING PING OK - Packet loss = 0%, RTA = 0.28 ms [20:09:48] 2013/07/25 20:08 OK z-dat-s2-b SMTP SMTP OK - 0.133 sec. response time [20:10:44] 2013/07/25 20:10 CRIT yucca CRITICAL - Host Unreachable (yucca) [20:13:44] 2013/07/25 20:12 CRIT z-dat-s2-b / Connection refused or timed out [20:13:44] 2013/07/25 20:12 CRIT z-dat-s2-b /sql Connection refused or timed out [20:13:45] 2013/07/25 20:12 CRIT z-dat-s2-b /tmp Connection refused or timed out [20:13:45] 2013/07/25 20:12 CRIT z-dat-s2-b Load avg. Connection refused or timed out [20:13:45] 2013/07/25 20:12 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (113) [20:13:45] 2013/07/25 20:12 CRIT z-dat-s2-b PING CRITICAL - Host Unreachable (z-dat-s2-b) [20:13:45] 2013/07/25 20:12 CRIT z-dat-s2-b SMTP No route to host [20:13:46] 2013/07/25 20:12 CRIT z-dat-s2-b SSH No route to host [20:15:45] 2013/07/25 20:15 CRIT z-dat-s2-b CRITICAL - Host Unreachable (z-dat-s2-b) [20:15:45] 2013/07/25 20:12 CRIT z-dat-s2-b APT Connection refused or timed out [20:59:47] 2013/07/25 20:59 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3493 [20:59:47] 2013/07/25 20:59 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3563.000000 [21:06:47] 2013/07/25 21:06 OK z-dat-s2-b PING OK - Packet loss = 0%, RTA = 0.30 ms [21:06:47] 2013/07/25 21:06 OK z-dat-s2-b / DISK OK - free space: / 2570 MB (67% inode=89%): [21:06:47] 2013/07/25 21:06 OK z-dat-s2-b /sql DISK OK - free space: /sql 152406 MB (15% inode=99%): [21:06:47] 2013/07/25 21:06 OK z-dat-s2-b Load avg. OK - load average: 0.95, 0.24, 0.08 [21:06:47] 2013/07/25 21:06 OK z-dat-s2-b MySQL Uptime: 37 Threads: 3 Questions: 1512 Slow queries: 0 Opens: 4095 Flush tables: 1 Open tables: 111 Queries per second avg: 40.864 [21:06:48] 2013/07/25 21:06 OK z-dat-s2-b SMTP SMTP OK - 0.117 sec. response time [21:07:47] 2013/07/25 21:06 OK z-dat-s2-b /tmp DISK OK - free space: / 2570 MB (67% inode=89%): [21:07:47] 2013/07/25 21:06 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [21:07:47] 2013/07/25 21:06 OK z-dat-s2-b PING PING OK - Packet loss = 0%, RTA = 0.18 ms [21:07:47] 2013/07/25 21:06 OK z-dat-s2-b SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [21:09:11] [[Admin:ToDo]] 10https://wiki.toolserver.org/w/index.php?diff=8094&oldid=8093&rcid=22096 * Nosy * (-42) (/* Databases */ ) [21:11:48] 2013/07/25 21:11 OK daphne MySQL slave Uptime: 8472300 Threads: 17 Questions: 1515600000 Slow queries: 1261434 Opens: 267135 Flush tables: 1 Open tables: 2151 Queries per second avg: 178.888 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1685 [21:11:48] 2013/07/25 21:11 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1725.000000