[06:47:55] 2013/11/04 06:47 CRIT rosemary /sql DISK CRITICAL - free space: /sql 18355 MB (5% inode=99%): [07:38:58] 2013/11/04 07:38 CRIT amaranth PING PING CRITICAL - Packet loss = 100% [07:38:58] 2013/11/04 07:38 CRIT amaranth SMTP CRITICAL - Socket timeout after 10 seconds [07:38:58] 2013/11/04 07:38 CRIT amaranth SSH CRITICAL - Socket timeout after 10 seconds [07:38:58] 2013/11/04 07:38 CRIT web.amaranth PING PING CRITICAL - Packet loss = 100% [07:38:58] 2013/11/04 07:37 CRIT web.amaranth fisheye.toolserver.org CRITICAL - Socket timeout after 21 seconds [07:38:58] 2013/11/04 07:38 CRIT web.amaranth wiki.toolserver.org CRITICAL - Socket timeout after 16 seconds [07:39:58] 2013/11/04 07:38 CRIT web.amaranth jira.toolserver.org CRITICAL - Socket timeout after 16 seconds [07:42:57] 2013/11/04 07:42 OK web.amaranth jira.toolserver.org HTTP OK: HTTP/1.1 200 OK - 521 bytes in 3.559 second response time [07:43:57] 2013/11/04 07:43 OK amaranth PING PING OK - Packet loss = 0%, RTA = 124.25 ms [07:43:57] 2013/11/04 07:43 OK amaranth SMTP SMTP OK - 0.786 sec. response time [07:43:58] 2013/11/04 07:43 OK amaranth SSH SSH OK - OpenSSH_5.6p1-hpn13v10 (protocol 2.0) [07:43:58] 2013/11/04 07:43 OK web.amaranth PING PING OK - Packet loss = 0%, RTA = 125.54 ms [07:43:58] 2013/11/04 07:42 OK web.amaranth fisheye.toolserver.org HTTP OK: HTTP/1.1 200 OK - 273 bytes in 9.582 second response time [07:43:58] 2013/11/04 07:43 OK web.amaranth wiki.toolserver.org HTTP OK: HTTP/1.1 200 OK - 349 bytes in 0.729 second response time [07:57:59] 2013/11/04 07:51 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2056 [08:35:02] 2013/11/04 08:34 CRIT z-dat-s6-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3634 [14:23:18] 2013/11/04 14:16 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2017 [14:41:19] 2013/11/04 14:40 OK z-dat-s2-b MySQL slave Uptime: 4335084 Threads: 37 Questions: 1659557692 Slow queries: 29289 Opens: 14898316 Flush tables: 1 Open tables: 256 Queries per second avg: 382.820 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1695 [14:44:19] 2013/11/04 14:37 WARN wolfsbane / DISK WARNING - free space: / 5610 MB (18% inode=93%): [14:44:20] 2013/11/04 14:37 WARN wolfsbane /tmp DISK WARNING - free space: / 5611 MB (18% inode=93%): [18:22:34] 2013/11/04 18:16 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1921.000000 [18:23:34] 2013/11/04 18:17 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1928 [18:40:34] 2013/11/04 18:40 OK daphne MySQL slave Uptime: 17276013 Threads: 188 Questions: 2551070438 Slow queries: 1766659 Opens: 501087 Flush tables: 1 Open tables: 2263 Queries per second avg: 147.665 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1788 [18:40:34] 2013/11/04 18:40 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1788.000000 [18:41:42] hey all, who are our toolserver admins [19:29:47] pleclown, you have any ideas who toolserver admins are? [19:32:02] mmmmh [19:32:10] nope [19:32:38] 2013/11/04 19:26 CRIT z-dat-s1-b MySQL Can't connect to MySQL server on 'z-dat-s1-b' (111) [19:33:50] we need to get this cropbot problem on commons fixed, and apparently toolserver admins can fix it on the spot [19:34:26] the admins are Nosy, DaB and Duesentrieb [19:34:55] but I don't know their usernames on IRC, or even if they idle here [19:35:22] hmm ok thanks [20:13:41] 2013/11/04 20:13 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3572 [20:35:42] 2013/11/04 20:35 OK z-dat-s6-a MySQL slave Uptime: 460457 Threads: 50 Questions: 56692487 Slow queries: 35684 Opens: 2497 Flush tables: 1 Open tables: 1035 Queries per second avg: 123.122 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1757 [22:54:51] 2013/11/04 22:48 CRIT rosemary MySQL Too many connections [22:56:50] 2013/11/04 22:56 OK z-dat-s1-b MySQL Uptime: 2256 Threads: 3 Questions: 3125 Slow queries: 1 Opens: 25 Flush tables: 1 Open tables: 13 Queries per second avg: 1.385 [22:56:50] 2013/11/04 22:55 WARN z-dat-s1-b MySQL slave No slaves defined [23:04:35] [[Special:Log/newusers]] create 10 * Katy655 * (New user account) [23:08:50] 2013/11/04 23:08 OK rosemary MySQL Uptime: 392686 Threads: 501 Questions: 129388374 Slow queries: 306259 Opens: 3549 Flush tables: 1 Open tables: 1109 Queries per second avg: 329.495 [23:16:51] 2013/11/04 23:10 CRIT rosemary MySQL Too many connections [23:52:53] 2013/11/04 23:52 OK rosemary MySQL Uptime: 395325 Threads: 501 Questions: 129436219 Slow queries: 311708 Opens: 3549 Flush tables: 1 Open tables: 1109 Queries per second avg: 327.417