[00:15:02] 2013/08/08 00:14 OK web.amaranth PING OK - Packet loss = 0%, RTA = 120.42 ms [00:15:02] 2013/08/08 00:14 WARN amaranth PING PING WARNING - Packet loss = 37%, RTA = 120.59 ms [00:15:03] 2013/08/08 00:00 CRIT cassia MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:15:03] 2013/08/08 00:00 CRIT daphne MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:15:03] 2013/08/08 00:00 CRIT rosemary MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:15:04] 2013/08/08 00:14 OK web.amaranth journal.toolserver.org HTTP OK: HTTP/1.1 200 OK - 213 bytes in 3.984 second response time [00:15:04] 2013/08/08 00:14 OK web.amaranth wiki.toolserver.org HTTP OK: HTTP/1.1 200 OK - 349 bytes in 0.848 second response time [00:15:05] 2013/08/08 00:00 CRIT z-dat-s1-b MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:15:05] 2013/08/08 00:01 CRIT z-dat-s2-b MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:15:06] 2013/08/08 00:00 CRIT z-dat-s3-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:15:06] 2013/08/08 00:00 CRIT z-dat-s4-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:15:07] 2013/08/08 00:00 CRIT z-dat-s5-b MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:15:08] 2013/08/08 00:00 CRIT z-dat-s6-a MySQL slave Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [00:16:03] 2013/08/08 00:15 OK amaranth PING OK - Packet loss = 0%, RTA = 122.08 ms [00:16:03] 2013/08/08 00:15 OK amaranth PING PING OK - Packet loss = 0%, RTA = 121.80 ms [00:16:03] 2013/08/08 00:14 OK amaranth SMTP SMTP OK - 0.621 sec. response time [00:16:03] 2013/08/08 00:15 OK amaranth SSH SSH OK - OpenSSH_5.6p1-hpn13v10 (protocol 2.0) [00:16:03] 2013/08/08 00:14 OK ha-dns-recursor.esi DNS recursor DNS OK: 0.049 seconds response time. www.google.com returns 173.194.66.103,173.194.66.104,173.194.66.105,173.194.66.106,173.194.66.147,173.194.66.99 [00:16:03] 2013/08/08 00:14 OK ha-proxy.esi HTTP proxy HTTP OK: HTTP/1.0 302 Moved Temporarily - 1134 bytes in 0.104 second response time [00:16:04] 2013/08/08 00:15 OK web.amaranth PING PING OK - Packet loss = 0%, RTA = 123.40 ms [00:16:04] 2013/08/08 00:14 OK web.amaranth fisheye.toolserver.org HTTP OK: HTTP/1.1 200 OK - 273 bytes in 8.793 second response time [00:16:05] 2013/08/08 00:15 OK web.amaranth jira.toolserver.org HTTP OK: HTTP/1.1 200 OK - 522 bytes in 0.542 second response time [00:20:03] 2013/08/08 00:13 WARN cassia s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2203.000000 [00:20:03] 2013/08/08 00:13 WARN cassia s5 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2199.000000 [00:20:03] 2013/08/08 00:13 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2165.000000 [00:20:03] 2013/08/08 00:13 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2184.000000 [00:20:03] 2013/08/08 00:13 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2176.000000 [00:20:04] 2013/08/08 00:13 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2181.000000 [00:21:04] 2013/08/08 00:13 WARN rosemary s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2206.000000 [00:44:05] 2013/08/08 00:43 CRIT cassia s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3643.000000 [00:44:05] 2013/08/08 00:43 CRIT cassia s5 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3639.000000 [00:44:05] 2013/08/08 00:43 CRIT daphne s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3605.000000 [00:44:05] 2013/08/08 00:43 CRIT rosemary s1 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3625.000000 [00:44:05] 2013/08/08 00:43 CRIT z-dat-s5-b s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3616.000000 [00:44:05] 2013/08/08 00:43 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3621.000000 [00:45:05] 2013/08/08 00:43 CRIT rosemary s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3706.000000 [00:47:11] [[Special:Log/newusers]] create 10 * Heather199 * (New user account) [01:59:08] 2013/08/08 01:57 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [02:00:08] 2013/08/08 01:58 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63467 MB (10% inode=99%): [02:33:23] Greetings. Just wantet to say that I get an error at https://toolserver.org/~luxo/contributions/contributions.php?user=201.15.116.176&blocks=true [02:33:34] An MySQL error [03:08:11] 2013/08/08 03:07 OK wolfsbane / DISK OK - free space: / 7157 MB (23% inode=93%): [03:08:11] 2013/08/08 03:07 OK wolfsbane /tmp DISK OK - free space: / 7157 MB (23% inode=93%): [04:21:14] 2013/08/08 04:14 WARN wolfsbane / DISK WARNING - free space: / 6181 MB (20% inode=93%): [04:22:15] 2013/08/08 04:15 WARN wolfsbane /tmp DISK WARNING - free space: / 6190 MB (20% inode=93%): [06:08:21] 2013/08/08 06:07 OK wolfsbane / DISK OK - free space: / 6866 MB (22% inode=93%): [06:08:21] 2013/08/08 06:07 OK wolfsbane /tmp DISK OK - free space: / 6879 MB (22% inode=93%): [09:44:35] 2013/08/08 09:37 WARN wolfsbane / DISK WARNING - free space: / 5422 MB (18% inode=93%): [09:44:36] 2013/08/08 09:38 WARN wolfsbane /tmp DISK WARNING - free space: / 5420 MB (18% inode=93%): [09:45:33] [[Special:Log/newusers]] create 10 * Travis791 * (New user account) [10:14:37] 2013/08/08 10:13 OK wolfsbane / DISK OK - free space: / 11494 MB (38% inode=93%): [10:14:37] 2013/08/08 10:14 OK wolfsbane /tmp DISK OK - free space: / 11488 MB (38% inode=93%): [10:21:36] 2013/08/08 10:20 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [10:22:36] 2013/08/08 10:21 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63139 MB (10% inode=99%): [10:22:37] 2013/08/08 10:21 OK z-dat-s7-a MySQL Uptime: 456 Threads: 9 Questions: 1755 Slow queries: 2 Opens: 133 Flush tables: 1 Open tables: 120 Queries per second avg: 3.848 [10:28:50] MySQL service seems to be down, does anyone around know about it? [10:32:37] 2013/08/08 10:32 WARN cassia MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3311 [10:33:37] 2013/08/08 10:32 WARN cassia s5 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3067.000000 [10:33:40] And now seems to be back [10:34:37] 2013/08/08 10:34 OK cassia MySQL slave Uptime: 4674520 Threads: 10 Questions: 3346962077 Slow queries: 68419 Opens: 16302 Flush tables: 1 Open tables: 14197 Queries per second avg: 716.1 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1399 [10:35:37] 2013/08/08 10:34 OK cassia s5 replag QUERY OK: 'SELECT ts_rc_age()' returned 1170.000000 [11:13:41] 2013/08/08 11:12 WARN ortelius Load avg. WARNING - load average: 12.59, 21.44, 12.63 [11:13:41] 2013/08/08 11:13 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3224 [11:16:40] 2013/08/08 11:15 OK ortelius Load avg. OK - load average: 4.21, 13.18, 10.88 [11:17:40] 2013/08/08 11:17 OK z-dat-s2-b MySQL slave Uptime: 317204 Threads: 9 Questions: 231064158 Slow queries: 4791 Opens: 2841739 Flush tables: 1 Open tables: 256 Queries per second avg: 728.440 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1510 [11:27:40] 2013/08/08 11:26 WARN cassia s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3086.000000 [11:30:41] 2013/08/08 11:29 OK cassia s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1611.000000 [11:31:41] 2013/08/08 11:30 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3345 [11:33:41] 2013/08/08 11:33 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3254 [11:33:41] 2013/08/08 11:33 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3371.000000 [11:34:41] 2013/08/08 11:33 OK z-dat-s6-a MySQL slave Uptime: 92493 Threads: 5 Questions: 11222322 Slow queries: 2800 Opens: 12534 Flush tables: 1 Open tables: 2886 Queries per second avg: 121.331 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1311 [11:37:41] 2013/08/08 11:37 OK daphne MySQL slave Uptime: 9647460 Threads: 18 Questions: 1822812235 Slow queries: 1323067 Opens: 306514 Flush tables: 1 Open tables: 2021 Queries per second avg: 188.942 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1502 [11:37:41] 2013/08/08 11:37 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1606.000000 [11:45:41] 2013/08/08 11:44 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3220 [11:48:42] 2013/08/08 11:47 OK z-dat-s4-a MySQL slave Uptime: 1128173 Threads: 2 Questions: 43208509 Slow queries: 27 Opens: 128 Flush tables: 1 Open tables: 95 Queries per second avg: 38.299 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1295 [11:48:42] 2013/08/08 11:48 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3280.000000 [11:50:42] 2013/08/08 11:49 WARN rosemary s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3405.000000 [11:51:42] 2013/08/08 11:51 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1684.000000 [11:55:42] 2013/08/08 11:54 OK rosemary s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1508.000000 [12:09:43] 2013/08/08 12:08 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3464 [12:09:43] 2013/08/08 12:08 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3464.000000 [12:27:44] 2013/08/08 12:26 OK z-dat-s5-b MySQL slave Uptime: 384265 Threads: 5 Questions: 315796078 Slow queries: 1742 Opens: 160419 Flush tables: 1 Open tables: 256 Queries per second avg: 821.818 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1632 [12:27:44] 2013/08/08 12:26 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1633.000000 [12:37:44] 2013/08/08 12:31 CRIT wolfsbane Load avg. CRITICAL - load average: 27.58, 29.08, 17.62 [12:39:44] 2013/08/08 12:39 WARN wolfsbane Load avg. WARNING - load average: 9.68, 22.10, 16.42 [12:42:45] 2013/08/08 12:42 OK wolfsbane Load avg. OK - load average: 4.98, 14.14, 14.24 [12:46:45] 2013/08/08 12:46 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3596 [12:51:46] 2013/08/08 12:51 OK z-dat-s3-a MySQL slave Uptime: 184782 Threads: 22 Questions: 257391108 Slow queries: 24160 Opens: 1424208 Flush tables: 1 Open tables: 16383 Queries per second avg: 1392.944 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1635 [12:55:46] 2013/08/08 12:55 WARN z-dat-s1-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3496 [12:58:15] [[Special:Log/newusers]] create 10 * Retta827 * (New user account) [13:06:46] 2013/08/08 13:06 OK z-dat-s1-b MySQL slave Uptime: 601301 Threads: 13 Questions: 294947750 Slow queries: 6063 Opens: 3316 Flush tables: 1 Open tables: 665 Queries per second avg: 490.515 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1665 [13:41:48] 2013/08/08 13:41 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3438 [13:45:47] 2013/08/08 13:45 OK z-dat-s7-a MySQL slave Uptime: 12662 Threads: 7 Questions: 7001431 Slow queries: 322 Opens: 1492 Flush tables: 1 Open tables: 1162 Queries per second avg: 552.948 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1616 [14:38:48] 2013/08/08 14:37 CRIT ortelius Load avg. CRITICAL - load average: 36.38, 21.36, 14.81 [14:52:48] 2013/08/08 14:51 WARN ortelius Load avg. WARNING - load average: 1.37, 12.80, 19.34 [14:57:49] 2013/08/08 14:56 OK ortelius Load avg. OK - load average: 1.46, 5.62, 14.29 [16:30:55] 2013/08/08 16:24 WARN willow /tmp DISK WARNING - free space: / 16595 MB (15% inode=98%): [16:31:55] 2013/08/08 16:24 WARN willow / DISK WARNING - free space: / 16403 MB (15% inode=98%): [16:37:55] 2013/08/08 16:36 CRIT willow / DISK CRITICAL - free space: / 11343 MB (10% inode=98%): [16:37:55] 2013/08/08 16:37 CRIT willow /tmp DISK CRITICAL - free space: / 10636 MB (10% inode=97%): [16:39:55] 2013/08/08 16:39 OK nightshade aliasd TCP OK - 0.006 second response time on port 984 [500 Not found.] [17:31:57] 2013/08/08 17:25 CRIT willow NTP CRITICAL - Socket timeout after 10 seconds [17:45:57] 2013/08/08 17:45 OK willow NTP NTP OK: Offset 0.000966 secs [17:54:56] 2013/08/08 17:54 WARN willow /tmp DISK WARNING - free space: / 12393 MB (11% inode=98%): [17:55:57] 2013/08/08 17:54 OK willow / DISK OK - free space: / 23844 MB (22% inode=99%): [17:55:57] 2013/08/08 17:55 OK willow /tmp DISK OK - free space: / 42309 MB (40% inode=99%): [18:21:58] 2013/08/08 18:21 OK yarrow aliasd TCP OK - 0.001 second response time on port 984 [500 Not found.] [18:43:01] 2013/08/08 18:36 WARN willow /tmp DISK WARNING - free space: / 16916 MB (16% inode=98%): [18:44:01] 2013/08/08 18:36 WARN willow / DISK WARNING - free space: / 16850 MB (16% inode=98%): [18:51:01] 2013/08/08 18:49 CRIT willow / DISK CRITICAL - free space: / 11456 MB (10% inode=98%): [18:51:01] 2013/08/08 18:50 CRIT willow /tmp DISK CRITICAL - free space: / 10820 MB (10% inode=97%): [18:56:02] 2013/08/08 18:54 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3573 [18:56:02] 2013/08/08 18:55 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3493.000000 [19:10:03] 2013/08/08 19:08 OK rosemary MySQL slave Uptime: 3232804 Threads: 8 Questions: 2250509521 Slow queries: 204437 Opens: 132162 Flush tables: 1 Open tables: 3575 Queries per second avg: 696.147 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1780 [19:10:04] 2013/08/08 19:09 OK rosemary s1 replag QUERY OK: 'SELECT ts_rc_age()' returned 1678.000000 [19:30:05] 2013/08/08 19:29 OK willow / DISK OK - free space: / 23299 MB (22% inode=99%): [19:30:05] 2013/08/08 19:29 OK willow /tmp DISK OK - free space: / 23440 MB (22% inode=99%): [19:39:06] 2013/08/08 19:32 WARN willow / DISK WARNING - free space: / 18045 MB (17% inode=98%): [19:39:06] 2013/08/08 19:32 WARN willow /tmp DISK WARNING - free space: / 18045 MB (17% inode=98%): [19:42:06] 2013/08/08 19:41 OK willow / DISK OK - free space: / 23507 MB (22% inode=99%): [19:42:06] 2013/08/08 19:41 OK willow /tmp DISK OK - free space: / 23647 MB (22% inode=99%): [20:37:07] 2013/08/08 20:36 CRIT hemlock / DISK CRITICAL - free space: / 2196 MB (10% inode=87%): [20:37:07] 2013/08/08 20:36 CRIT hemlock /tmp DISK CRITICAL - free space: / 2196 MB (10% inode=87%): [20:43:08] 2013/08/08 20:36 WARN hemlock /aux0 DISK WARNING - free space: / 2195 MB (10% inode=87%): [23:26:03] [[Special:Log/newusers]] create 10 * Quintin284 * (New user account)