[00:01:18] 2013/11/09 00:00 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:04:18] 2013/11/09 00:03 CRIT z-dat-s4-a /sql DISK CRITICAL - free space: /sql 0 MB (0% inode=26%): [00:06:18] 2013/11/09 00:05 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:07:19] 2013/11/09 00:06 CRIT z-dat-s4-a /sql DISK CRITICAL - free space: /sql 0 MB (0% inode=26%): [00:11:19] 2013/11/09 00:10 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:19] 2013/11/09 00:10 ?? hyacinth / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:19] 2013/11/09 00:10 ?? hyacinth /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:19] 2013/11/09 00:10 ?? hyacinth Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:19] 2013/11/09 00:10 ?? hyacinth Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:19] 2013/11/09 00:10 ?? hyacinth RAID CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:20] 2013/11/09 00:10 CRIT hyacinth SSH CRITICAL - Socket timeout after 10 seconds [00:17:20] 2013/11/09 00:10 ?? z-dat-s3-a / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:21] 2013/11/09 00:10 ?? z-dat-s3-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:21] 2013/11/09 00:10 ?? z-dat-s3-a /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:22] 2013/11/09 00:10 ?? z-dat-s3-a Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:22] 2013/11/09 00:10 ?? z-dat-s4-a / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:23] 2013/11/09 00:10 ?? z-dat-s4-a /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:23] 2013/11/09 00:10 ?? z-dat-s4-a Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:17:24] 2013/11/09 00:10 ?? z-dat-s6-a / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:19:19] 2013/11/09 00:18 OK hyacinth / DISK OK - free space: / 9008 MB (30% inode=86%): [00:19:19] 2013/11/09 00:18 OK hyacinth /tmp DISK OK - free space: /tmp 1960 MB (99% inode=99%): [00:19:19] 2013/11/09 00:18 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [00:19:19] 2013/11/09 00:18 OK hyacinth Load avg. OK - load average: 2.60, 3.27, 3.30 [00:19:19] 2013/11/09 00:18 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:19:20] 2013/11/09 00:18 OK hyacinth SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:19:20] 2013/11/09 00:18 OK z-dat-s3-a / DISK OK - free space: / 9008 MB (30% inode=86%): [00:19:21] 2013/11/09 00:18 OK z-dat-s3-a /sql DISK OK - free space: /sql 157026 MB (16% inode=99%): [00:19:21] 2013/11/09 00:18 OK z-dat-s3-a /tmp DISK OK - free space: /tmp 1997 MB (99% inode=99%): [00:19:22] 2013/11/09 00:18 OK z-dat-s3-a Load avg. OK - load average: 2.41, 3.18, 3.26 [00:19:22] 2013/11/09 00:18 OK z-dat-s4-a / DISK OK - free space: / 9008 MB (30% inode=86%): [00:19:23] 2013/11/09 00:18 CRIT z-dat-s4-a /sql DISK CRITICAL - free space: /sql 0 MB (0% inode=26%): [00:19:23] 2013/11/09 00:18 OK z-dat-s4-a /tmp DISK OK - free space: /tmp 1995 MB (99% inode=99%): [00:19:24] 2013/11/09 00:18 OK z-dat-s4-a Load avg. OK - load average: 3.97, 3.55, 3.39 [00:24:20] 2013/11/09 00:17 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [00:24:20] 2013/11/09 00:17 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [00:34:20] 2013/11/09 00:33 OK z-dat-s3-a MySQL Uptime: 964 Threads: 12 Questions: 2039 Slow queries: 1 Opens: 184 Flush tables: 1 Open tables: 173 Queries per second avg: 2.115 [00:34:20] 2013/11/09 00:33 OK z-dat-s3-a MySQL slave Uptime: 964 Threads: 12 Questions: 2040 Slow queries: 1 Opens: 184 Flush tables: 1 Open tables: 173 Queries per second avg: 2.116 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 946 [01:41:22] 2013/11/09 01:34 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [01:41:22] 2013/11/09 01:34 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [01:56:24] 2013/11/09 01:55 OK z-dat-s3-a MySQL Uptime: 1258 Threads: 9 Questions: 3026 Slow queries: 1 Opens: 172 Flush tables: 1 Open tables: 151 Queries per second avg: 2.405 [01:56:24] 2013/11/09 01:55 OK z-dat-s3-a MySQL slave Uptime: 1258 Threads: 9 Questions: 3026 Slow queries: 1 Opens: 172 Flush tables: 1 Open tables: 151 Queries per second avg: 2.405 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1375 [06:55:50] [[Special:Log/newusers]] create 10 * Kathey121 * (New user account) [12:26:02] 2013/11/09 12:25 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:27:02] 2013/11/09 12:26 CRIT z-dat-s4-a /sql DISK CRITICAL - free space: /sql 0 MB (0% inode=26%): [12:53:04] 2013/11/09 12:52 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:05] 2013/11/09 12:52 ?? hyacinth /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:05] 2013/11/09 12:54 ?? hyacinth Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:05] 2013/11/09 12:52 ?? hyacinth RAID CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:05] 2013/11/09 12:52 ?? z-dat-s3-a / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:05] 2013/11/09 12:52 ?? z-dat-s3-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:06] 2013/11/09 12:52 ?? z-dat-s3-a /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:06] 2013/11/09 12:52 ?? z-dat-s3-a Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:07] 2013/11/09 12:52 ?? z-dat-s4-a /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:07] 2013/11/09 12:52 ?? z-dat-s4-a Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:08] 2013/11/09 12:52 ?? z-dat-s6-a / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:08] 2013/11/09 12:52 ?? z-dat-s6-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:09] 2013/11/09 12:52 ?? z-dat-s6-a /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:09] 2013/11/09 12:52 ?? z-dat-s7-a / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:59:10] 2013/11/09 12:52 ?? z-dat-s7-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:00:05] 2013/11/09 12:59 OK hyacinth /tmp DISK OK - free space: /tmp 2007 MB (99% inode=99%): [13:00:05] 2013/11/09 12:59 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [13:00:05] 2013/11/09 12:59 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:00:05] 2013/11/09 12:59 OK z-dat-s3-a / DISK OK - free space: / 9007 MB (30% inode=86%): [13:00:05] 2013/11/09 12:59 OK z-dat-s3-a /sql DISK OK - free space: /sql 156721 MB (16% inode=99%): [13:00:06] 2013/11/09 12:59 OK z-dat-s3-a /tmp DISK OK - free space: /tmp 1997 MB (99% inode=99%): [13:00:06] 2013/11/09 12:59 OK z-dat-s3-a Load avg. OK - load average: 1.89, 2.46, 2.80 [13:00:07] 2013/11/09 12:59 CRIT z-dat-s4-a /sql DISK CRITICAL - free space: /sql 0 MB (0% inode=26%): [13:00:07] 2013/11/09 12:59 OK z-dat-s4-a /tmp DISK OK - free space: /tmp 1995 MB (99% inode=99%): [13:00:08] 2013/11/09 12:59 OK z-dat-s4-a Load avg. OK - load average: 1.87, 2.45, 2.80 [13:00:08] 2013/11/09 12:59 OK z-dat-s6-a / DISK OK - free space: / 9007 MB (30% inode=86%): [13:00:09] 2013/11/09 12:59 OK z-dat-s6-a /sql DISK OK - free space: /sql 280531 MB (34% inode=99%): [13:00:09] 2013/11/09 12:59 OK z-dat-s6-a /tmp DISK OK - free space: /tmp 2004 MB (99% inode=99%): [13:00:10] 2013/11/09 12:59 OK z-dat-s7-a / DISK OK - free space: / 9007 MB (30% inode=86%): [13:06:05] 2013/11/09 12:58 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [13:06:06] 2013/11/09 12:58 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [13:20:06] 2013/11/09 13:18 OK z-dat-s3-a MySQL Uptime: 1225 Threads: 10 Questions: 281 Slow queries: 2 Opens: 91 Flush tables: 1 Open tables: 80 Queries per second avg: 0.229 [13:20:06] 2013/11/09 13:18 OK z-dat-s3-a MySQL slave Uptime: 1225 Threads: 10 Questions: 281 Slow queries: 2 Opens: 91 Flush tables: 1 Open tables: 80 Queries per second avg: 0.229 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1230 [13:30:03] [[MySQL queries]] ! 10https://wiki.toolserver.org/w/index.php?diff=8212&oldid=8197&rcid=22429 * Staeiou * (+187) (most edited pages) [13:40:07] 2013/11/09 13:33 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1957 [13:40:07] 2013/11/09 13:33 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1949.000000 [15:00:11] 2013/11/09 14:59 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3614 [15:00:11] 2013/11/09 14:59 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3606.000000 [15:01:11] 2013/11/09 15:00 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3596.000000 [15:02:11] 2013/11/09 15:01 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3560 [15:48:13] 2013/11/09 15:47 OK z-dat-s5-b MySQL slave Uptime: 1997836 Threads: 3 Questions: 2651108407 Slow queries: 14623 Opens: 59900 Flush tables: 1 Open tables: 256 Queries per second avg: 1326.990 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1747 [15:48:13] 2013/11/09 15:47 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1744.000000 [15:58:15] 2013/11/09 15:51 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1972 [15:58:15] 2013/11/09 15:51 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1965.000000 [16:02:14] 2013/11/09 16:01 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1799.000000 [16:03:14] 2013/11/09 16:02 OK z-dat-s5-b MySQL slave Uptime: 1998736 Threads: 3 Questions: 2651376542 Slow queries: 14632 Opens: 60073 Flush tables: 1 Open tables: 256 Queries per second avg: 1326.526 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1722 [16:29:16] 2013/11/09 16:24 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (110) [16:29:16] 2013/11/09 16:25 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (110) [16:31:16] 2013/11/09 16:24 CRIT hyacinth RAID CHECK_NRPE: Socket timeout after 30 seconds. [16:31:16] 2013/11/09 16:26 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [16:32:16] 2013/11/09 16:25 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [16:32:17] 2013/11/09 16:23 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [16:34:17] 2013/11/09 16:26 CRIT z-dat-s6-a MySQL slave Can't connect to MySQL server on 'z-dat-s6-a' (110) [16:35:16] 2013/11/09 16:34 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:35:16] 2013/11/09 16:33 OK z-dat-s4-a MySQL Uptime: 5725734 Threads: 1 Questions: 341893057 Slow queries: 163 Opens: 527 Flush tables: 1 Open tables: 251 Queries per second avg: 59.711 [16:35:16] 2013/11/09 16:34 OK z-dat-s7-a MySQL Uptime: 2531111 Threads: 13 Questions: 398500033 Slow queries: 625774 Opens: 908145 Flush tables: 1 Open tables: 3493 Queries per second avg: 157.440 [16:35:16] 2013/11/09 16:34 OK z-dat-s7-a MySQL slave Uptime: 2531121 Threads: 15 Questions: 398518381 Slow queries: 625775 Opens: 909464 Flush tables: 1 Open tables: 3493 Queries per second avg: 157.447 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 570 [16:36:16] 2013/11/09 16:35 OK hyacinth SMTP SMTP OK - 0.002 sec. response time [16:36:16] 2013/11/09 16:34 OK z-dat-s6-a MySQL Uptime: 105586 Threads: 11 Questions: 16352773 Slow queries: 8188 Opens: 7627 Flush tables: 1 Open tables: 851 Queries per second avg: 154.876 [16:36:17] 2013/11/09 16:34 OK z-dat-s6-a MySQL slave Uptime: 105586 Threads: 12 Questions: 16352770 Slow queries: 8188 Opens: 7627 Flush tables: 1 Open tables: 851 Queries per second avg: 154.876 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 433 [21:54:32] 2013/11/09 21:48 WARN ptolemy Load avg. WARNING - load average: 18.14, 15.53, 9.65 [22:00:33] 2013/11/09 22:00 OK ptolemy Load avg. OK - load average: 11.39, 14.76, 11.45 [22:22:35] 2013/11/09 22:22 CRIT z-dat-s4-a Load avg. Connection refused or timed out [22:22:35] 2013/11/09 22:22 CRIT z-dat-s4-a MySQL Can't connect to MySQL server on 'z-dat-s4-a' (113) [22:23:35] 2013/11/09 22:22 CRIT z-dat-s4-a / Connection refused or timed out [22:23:35] 2013/11/09 22:22 CRIT z-dat-s4-a /tmp Connection refused or timed out [22:23:35] 2013/11/09 22:22 CRIT z-dat-s4-a PING CRITICAL - Host Unreachable (z-dat-s4-a) [22:33:35] 2013/11/09 22:32 CRIT z-dat-s4-a CRITICAL - Host Unreachable (z-dat-s4-a) [23:49:39] 2013/11/09 23:42 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1999 [23:49:39] 2013/11/09 23:43 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2031.000000