[02:51:17] 2013/08/21 02:44 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2057.000000 [03:25:19] 2013/08/21 03:24 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1685.000000 [04:30:20] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=8116&oldid=8110&rcid=22181 * 185.15.59.202 * (+0) (updated page) [06:33:29] 2013/08/21 06:27 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [08:50:41] 2013/08/21 08:50 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [10:36:44] 2013/08/21 10:36 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 55649 MB (5% inode=99%): [10:39:44] 2013/08/21 10:39 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 59575 MB (6% inode=99%): [11:15:49] 2013/08/21 11:09 WARN willow /tmp DISK WARNING - free space: / 15900 MB (15% inode=98%): [11:16:49] 2013/08/21 11:09 WARN willow / DISK WARNING - free space: / 15664 MB (14% inode=98%): [11:19:49] 2013/08/21 11:19 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 55936 MB (5% inode=99%): [11:20:49] 2013/08/21 11:20 CRIT willow /tmp DISK CRITICAL - free space: / 11295 MB (10% inode=98%): [11:21:49] 2013/08/21 11:20 CRIT willow / DISK CRITICAL - free space: / 10923 MB (10% inode=97%): [11:23:50] 2013/08/21 11:23 WARN willow /tmp DISK WARNING - free space: / 11762 MB (11% inode=98%): [11:23:50] 2013/08/21 11:23 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 59329 MB (6% inode=99%): [11:24:50] 2013/08/21 11:23 WARN willow / DISK WARNING - free space: / 11762 MB (11% inode=98%): [11:38:50] 2013/08/21 11:38 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 56227 MB (5% inode=99%): [11:39:50] 2013/08/21 11:39 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 59497 MB (6% inode=99%): [11:42:50] 2013/08/21 11:38 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.418 second response time [11:44:50] 2013/08/21 11:44 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.142 second response time [12:01:51] 2013/08/21 12:01 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 56183 MB (5% inode=99%): [12:02:51] 2013/08/21 12:02 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 56302 MB (6% inode=99%): [12:27:53] 2013/08/21 12:26 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.707 second response time [12:32:54] 2013/08/21 12:32 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.726 second response time [12:33:54] 2013/08/21 12:33 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.471 second response time [12:35:55] 2013/08/21 12:35 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.265 second response time [13:02:57] 2013/08/21 13:02 OK willow /tmp DISK OK - free space: / 41499 MB (39% inode=99%): [13:03:57] 2013/08/21 13:02 OK willow / DISK OK - free space: / 41499 MB (39% inode=99%): [17:48:09] 2013/08/21 17:47 WARN hemlock / DISK WARNING - free space: / 2945 MB (14% inode=87%): [17:48:09] 2013/08/21 17:47 OK hemlock /aux0 DISK OK - free space: / 2942 MB (14% inode=87%): [17:48:09] 2013/08/21 17:47 WARN hemlock /tmp DISK WARNING - free space: / 2949 MB (14% inode=87%): [17:49:09] 2013/08/21 17:48 OK hemlock / DISK OK - free space: / 6985 MB (34% inode=88%): [17:49:09] 2013/08/21 17:48 OK hemlock /tmp DISK OK - free space: / 6988 MB (34% inode=88%): [19:42:16] 2013/08/21 19:38 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2019 [19:45:16] 2013/08/21 19:37 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2215 [19:47:16] 2013/08/21 19:46 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:48:16] 2013/08/21 19:47 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 62736 MB (10% inode=99%): [19:48:17] 2013/08/21 19:41 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2145 [19:49:16] 2013/08/21 19:47 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (111) [19:51:16] 2013/08/21 19:49 CRIT z-dat-s3-a MySQL slave (Service Check Timed Out) [19:51:16] 2013/08/21 19:49 CRIT z-dat-s6-a MySQL slave (Service Check Timed Out) [19:53:16] 2013/08/21 19:49 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [19:53:16] 2013/08/21 19:48 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [19:54:16] 2013/08/21 19:50 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [19:54:16] 2013/08/21 19:49 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [19:54:16] 2013/08/21 19:47 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (111) [19:56:16] 2013/08/21 19:49 CRIT hyacinth Environment IPMI Timeout while attempting connection [19:56:16] 2013/08/21 19:49 CRIT hyacinth Load avg. Timeout while attempting connection [19:56:16] 2013/08/21 19:49 CRIT hyacinth NTP CRITICAL - Socket timeout after 10 seconds [19:56:16] 2013/08/21 19:49 CRIT hyacinth RAID Timeout while attempting connection [19:56:16] 2013/08/21 19:49 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [19:56:16] 2013/08/21 19:49 CRIT hyacinth SSH CRITICAL - Socket timeout after 10 seconds [19:56:17] 2013/08/21 19:49 CRIT z-dat-s4-a Load avg. CHECK_NRPE: Socket timeout after 30 seconds. [19:56:18] 2013/08/21 19:49 CRIT z-dat-s6-a Load avg. CHECK_NRPE: Socket timeout after 30 seconds. [19:57:16] 2013/08/21 19:56 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [19:57:16] 2013/08/21 19:56 OK hyacinth Load avg. OK - load average: 0.06, 0.22, 0.54 [19:57:16] 2013/08/21 19:56 OK hyacinth NTP NTP OK: Offset 8.5e-05 secs [19:57:16] 2013/08/21 19:56 OK z-dat-s3-a MySQL Uptime: 104400 Threads: 24 Questions: 78972919 Slow queries: 4437 Opens: 484458 Flush tables: 1 Open tables: 16377 Queries per second avg: 756.445 [19:57:16] 2013/08/21 19:56 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2905 [19:57:16] 2013/08/21 19:56 OK z-dat-s4-a Load avg. OK - load average: 0.05, 0.22, 0.55 [19:57:17] 2013/08/21 19:56 OK z-dat-s4-a MySQL Uptime: 2280717 Threads: 3 Questions: 95259449 Slow queries: 47 Opens: 130 Flush tables: 1 Open tables: 97 Queries per second avg: 41.767 [19:57:17] 2013/08/21 19:56 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2288 [19:57:18] 2013/08/21 19:56 OK z-dat-s6-a Load avg. OK - load average: 0.05, 0.22, 0.55 [19:57:18] 2013/08/21 19:56 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2678 [19:58:16] 2013/08/21 19:57 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:58:16] 2013/08/21 19:57 OK hyacinth SMTP SMTP OK - 0.124 sec. response time [19:58:16] 2013/08/21 19:57 OK hyacinth SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:58:16] 2013/08/21 19:57 OK z-dat-s6-a MySQL Uptime: 640928 Threads: 5 Questions: 124278382 Slow queries: 25343 Opens: 52277 Flush tables: 1 Open tables: 3095 Queries per second avg: 193.903 [20:04:16] 2013/08/21 19:57 CRIT yarrow aliasd Connection refused [20:09:17] 2013/08/21 20:08 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3609 [20:15:17] 2013/08/21 20:14 CRIT z-dat-s6-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3635 [20:25:18] 2013/08/21 20:24 CRIT z-dat-s4-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3649 [20:31:19] 2013/08/21 20:30 OK z-dat-s7-a MySQL Uptime: 2587 Threads: 5 Questions: 74 Slow queries: 0 Opens: 37 Flush tables: 1 Open tables: 27 Queries per second avg: 0.28 [21:35:30] nacht ts [21:50:23] 2013/08/21 21:49 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3489 [21:50:23] 2013/08/21 21:49 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2999 [21:54:23] 2013/08/21 21:53 OK z-dat-s6-a MySQL slave Uptime: 647916 Threads: 8 Questions: 124784676 Slow queries: 28741 Opens: 52293 Flush tables: 1 Open tables: 3111 Queries per second avg: 192.593 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1142 [21:58:24] 2013/08/21 21:57 OK z-dat-s4-a MySQL slave Uptime: 2287963 Threads: 1 Questions: 95702109 Slow queries: 50 Opens: 130 Flush tables: 1 Open tables: 97 Queries per second avg: 41.828 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1501 [22:13:23] 2013/08/21 22:12 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 56109 MB (5% inode=99%): [22:14:23] 2013/08/21 22:13 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 57373 MB (6% inode=99%): [22:30:25] 2013/08/21 22:28 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3547 [22:43:26] 2013/08/21 22:42 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3255 [22:46:27] 2013/08/21 22:45 OK z-dat-s3-a MySQL slave Uptime: 114542 Threads: 13 Questions: 86327788 Slow queries: 7490 Opens: 547089 Flush tables: 1 Open tables: 16374 Queries per second avg: 753.678 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [22:46:27] 2013/08/21 22:45 OK z-dat-s7-a MySQL slave Uptime: 10716 Threads: 2 Questions: 1759050 Slow queries: 399 Opens: 832 Flush tables: 1 Open tables: 803 Queries per second avg: 164.151 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1769 [22:48:26] 2013/08/21 22:47 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 56097 MB (5% inode=99%): [22:50:26] 2013/08/21 22:49 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 57490 MB (6% inode=99%):