[01:52:59] 2013/08/01 01:51 OK z-dat-s1-b MySQL Uptime: 8225 Threads: 5 Questions: 5763 Slow queries: 0 Opens: 71 Flush tables: 1 Open tables: 63 Queries per second avg: 0.700 [02:21:02] 2013/08/01 02:20 OK nightshade aliasd TCP OK - 0.003 second response time on port 984 [500 Not found.] [04:12:07] 2013/08/01 04:05 CRIT z-dat-s1-b MySQL Too many connections [04:15:07] 2013/08/01 04:13 WARN ortelius Load avg. WARNING - load average: 12.13, 17.64, 10.72 [04:17:07] 2013/08/01 04:16 OK ortelius Load avg. OK - load average: 4.03, 12.79, 9.74 [09:32:30] 2013/08/01 09:32 OK yucca PING OK - Packet loss = 0%, RTA = 0.12 ms [09:32:31] 2013/08/01 09:31 OK yucca /tmp DISK OK - free space: /tmp 4317 MB (96% inode=99%): [09:32:31] 2013/08/01 09:31 OK yucca APT APT OK: 0 packages available for upgrade (0 critical updates). [09:32:31] 2013/08/01 09:31 OK yucca Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [09:32:31] 2013/08/01 09:31 WARN yucca PING PING WARNING - Packet loss = 50%, RTA = 0.17 ms [09:32:31] 2013/08/01 09:31 OK yucca SMTP SMTP OK - 0.136 sec. response time [09:32:31] 2013/08/01 09:32 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [09:32:32] 2013/08/01 09:31 CRIT z-dat-s2-b APT Timeout while attempting connection [09:33:30] 2013/08/01 09:32 OK yucca / DISK OK - free space: / 1401 MB (78% inode=89%): [09:33:31] 2013/08/01 09:32 OK yucca Load avg. OK - load average: 0.04, 0.03, 0.05 [09:33:31] 2013/08/01 09:32 OK yucca PING PING OK - Packet loss = 0%, RTA = 0.21 ms [09:33:31] 2013/08/01 09:32 CRIT z-dat-s2-b / Connection refused or timed out [09:33:31] 2013/08/01 09:32 CRIT z-dat-s2-b /sql Connection refused or timed out [09:33:31] 2013/08/01 09:32 CRIT z-dat-s2-b /tmp Connection refused or timed out [09:33:31] 2013/08/01 09:32 CRIT z-dat-s2-b Load avg. Timeout while attempting connection [09:33:32] 2013/08/01 09:32 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (110) [09:33:32] 2013/08/01 09:32 CRIT z-dat-s2-b MySQL slave Can't connect to MySQL server on 'z-dat-s2-b' (113) [09:33:33] 2013/08/01 09:32 CRIT z-dat-s2-b PING CRITICAL - Host Unreachable (z-dat-s2-b) [09:33:33] 2013/08/01 09:32 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [09:33:34] 2013/08/01 09:32 CRIT z-dat-s2-b SSH No route to host [09:35:31] 2013/08/01 09:35 CRIT z-dat-s2-b CRITICAL - Host Unreachable (z-dat-s2-b) [09:35:31] 2013/08/01 09:34 CRIT yucca /tmp Timeout while attempting connection [09:35:31] 2013/08/01 09:34 CRIT yucca APT Timeout while attempting connection [09:35:31] 2013/08/01 09:34 CRIT yucca Environment IPMI Timeout while attempting connection [09:35:31] 2013/08/01 09:34 CRIT yucca Load avg. Timeout while attempting connection [09:35:31] 2013/08/01 09:34 CRIT yucca PING PING CRITICAL - Packet loss = 100% [09:35:31] 2013/08/01 09:34 CRIT yucca SMTP CRITICAL - Socket timeout after 10 seconds [09:35:32] 2013/08/01 09:35 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [09:36:31] 2013/08/01 09:35 CRIT yucca / Connection refused or timed out [09:37:31] 2013/08/01 09:36 CRIT yucca CRITICAL - Host Unreachable (yucca) [09:37:31] 2013/08/01 09:36 OK z-dat-s2-b PING OK - Packet loss = 0%, RTA = 0.15 ms [09:37:31] 2013/08/01 09:36 OK z-dat-s2-b /tmp DISK OK - free space: / 2570 MB (67% inode=89%): [09:37:31] 2013/08/01 09:36 WARN z-dat-s2-b APT APT WARNING: 34 packages available for upgrade (0 critical updates). [09:37:31] 2013/08/01 09:36 OK z-dat-s2-b MySQL slave Uptime: 158473 Threads: 4 Questions: 58739443 Slow queries: 1354 Opens: 683880 Flush tables: 1 Open tables: 255 Queries per second avg: 370.658 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [09:37:32] 2013/08/01 09:36 OK z-dat-s2-b PING PING OK - Packet loss = 0%, RTA = 0.16 ms [09:37:32] 2013/08/01 09:36 OK z-dat-s2-b SMTP SMTP OK - 1.041 sec. response time [09:37:32] 2013/08/01 09:36 OK z-dat-s2-b SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [09:38:31] 2013/08/01 09:37 OK z-dat-s2-b / DISK OK - free space: / 2570 MB (67% inode=89%): [09:38:31] 2013/08/01 09:37 OK z-dat-s2-b /sql DISK OK - free space: /sql 157103 MB (16% inode=99%): [09:38:31] 2013/08/01 09:37 OK z-dat-s2-b Load avg. OK - load average: 0.85, 1.45, 2.14 [09:38:31] 2013/08/01 09:37 OK z-dat-s2-b MySQL Uptime: 158525 Threads: 6 Questions: 58741480 Slow queries: 1354 Opens: 683913 Flush tables: 1 Open tables: 256 Queries per second avg: 370.550 [10:29:33] 2013/08/01 10:22 CRIT nightshade aliasd Connection refused [10:35:34] 2013/08/01 10:34 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:36:34] 2013/08/01 10:35 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 64088 MB (10% inode=99%): [10:39:35] 2013/08/01 10:35 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [10:39:35] 2013/08/01 10:35 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [10:41:35] 2013/08/01 10:34 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (111) [10:41:35] 2013/08/01 10:34 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (111) [10:43:35] 2013/08/01 10:42 OK z-dat-s7-a MySQL Uptime: 460 Threads: 2 Questions: 3248 Slow queries: 0 Opens: 201 Flush tables: 1 Open tables: 192 Queries per second avg: 7.60 [10:43:36] 2013/08/01 10:42 OK z-dat-s7-a MySQL slave Uptime: 460 Threads: 2 Questions: 3247 Slow queries: 0 Opens: 201 Flush tables: 1 Open tables: 192 Queries per second avg: 7.58 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1620 [10:57:36] 2013/08/01 10:50 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1997 [11:01:35] 2013/08/01 11:00 OK z-dat-s3-a MySQL Uptime: 1492 Threads: 17 Questions: 278 Slow queries: 0 Opens: 108 Flush tables: 1 Open tables: 100 Queries per second avg: 0.186 [11:01:35] 2013/08/01 11:00 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2966 [11:02:35] 2013/08/01 11:01 OK z-dat-s6-a MySQL slave Uptime: 93909 Threads: 9 Questions: 30852902 Slow queries: 3331 Opens: 12717 Flush tables: 1 Open tables: 2937 Queries per second avg: 328.540 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1781 [11:05:35] 2013/08/01 10:57 CRIT z-dat-s5-b s4 replag (Service Check Timed Out) [11:06:35] 2013/08/01 10:58 CRIT z-dat-s5-b wikidata replag (Service Check Timed Out) [11:09:36] 2013/08/01 11:08 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1907.000000 [11:10:36] 2013/08/01 11:07 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2013 [11:10:36] 2013/08/01 11:09 CRIT z-dat-s5-b wikidata replag (Service Check Timed Out) [11:17:37] 2013/08/01 11:16 CRIT z-dat-s5-b MySQL slave (Service Check Timed Out) [11:18:37] 2013/08/01 11:18 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2506 [11:37:37] 2013/08/01 11:37 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3646 [11:39:37] 2013/08/01 11:39 OK z-dat-s3-a MySQL slave Uptime: 3831 Threads: 23 Questions: 1361657 Slow queries: 46 Opens: 21935 Flush tables: 1 Open tables: 12885 Queries per second avg: 355.431 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1745 [12:17:40] 2013/08/01 12:15 WARN ptolemy Load avg. WARNING - load average: 25.29, 23.14, 13.53 [12:45:41] 2013/08/01 12:44 OK ptolemy Load avg. OK - load average: 5.81, 11.52, 14.76 [13:13:41] 2013/08/01 13:13 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 56112 MB (5% inode=99%): [14:01:42] 2013/08/01 14:01 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 64053 MB (6% inode=99%): [14:05:43] 2013/08/01 14:05 OK z-dat-s1-b MySQL Uptime: 40 Threads: 14 Questions: 123 Slow queries: 5 Opens: 623 Flush tables: 1 Open tables: 45 Queries per second avg: 3.75 [15:04:12] @replag [15:04:13] Merlissimo: s1-rr-a: 17h 10m 30s [+0.10 s/s]; s1-rr-a-wd: 2w 2d 10h 35m 40s [+8.91 s/s]; s1-user: 1m 37s [-0.00 s/s]; s1-user-wd: 18h 39m 52s [-0.18 s/s]; s2-user-c: error; s2-user-wd: 2d 19h 23m 43s [+1.00 s/s]; s3-user: 1m 7s [-0.00 s/s]; s3-user-wd: 2d 19h 23m 43s [+1.00 s/s] [15:04:14] Merlissimo: s4-user-wd: 18h 39m 52s [-1.89 s/s]; s5-rr-a-wd: 18h 39m 52s [-0.18 s/s]; s5-user: 2h 59m 47s [+0.06 s/s]; s5-user-c: 1h 20m 41s [+0.03 s/s]; s6-user-wd: 2d 19h 23m 43s [+1.00 s/s]; s7-user-wd: 2d 19h 23m 43s [+1.00 s/s] [15:22:48] 2013/08/01 15:22 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3533.000000 [15:59:52] 2013/08/01 15:59 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1782.000000 [16:53:55] 2013/08/01 16:53 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3329 [16:53:55] 2013/08/01 16:53 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3518.000000 [17:01:56] 2013/08/01 17:01 OK z-dat-s5-b MySQL slave Uptime: 1744623 Threads: 7 Questions: 2300681288 Slow queries: 16703 Opens: 1510782 Flush tables: 1 Open tables: 255 Queries per second avg: 1318.726 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1595 [17:01:56] 2013/08/01 17:01 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1729.000000 [19:37:31] https://jira.toolserver.org/browse/ACCAPP-664 [20:08:10] 2013/08/01 20:07 OK nightshade aliasd TCP OK - 0.001 second response time on port 984 [500 Not found.] [20:21:10] 2013/08/01 20:20 OK z-dat-s1-b MySQL slave Uptime: 22541 Threads: 11 Questions: 8957592 Slow queries: 482 Opens: 2021 Flush tables: 1 Open tables: 570 Queries per second avg: 397.391 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [20:31:10] 2013/08/01 20:24 CRIT z-dat-s1-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75364