[03:18:26] 2013/09/01 03:14 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1933 [03:21:25] 2013/09/01 03:14 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2033.000000 [04:08:28] 2013/09/01 04:07 OK z-dat-s5-b MySQL slave Uptime: 2427934 Threads: 8 Questions: 2834411444 Slow queries: 12531 Opens: 1195888 Flush tables: 1 Open tables: 256 Queries per second avg: 1167.417 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1484 [04:08:28] 2013/09/01 04:07 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1484.000000 [08:40:50] is it possible that the smtp server has stopped working since last september? [08:42:42] 2013/09/01 08:38 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [08:43:42] 2013/09/01 08:42 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.607 second response time [08:46:42] 2013/09/01 08:45 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 6.736 second response time [08:47:43] 2013/09/01 08:46 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.186 second response time [09:20:45] 2013/09/01 09:18 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.846 second response time [09:21:45] 2013/09/01 09:21 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.419 second response time [09:28:46] 2013/09/01 09:28 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.566 second response time [09:29:46] 2013/09/01 09:29 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [10:27:48] 2013/09/01 10:23 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [10:27:48] 2013/09/01 10:23 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [10:28:48] 2013/09/01 10:24 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [10:28:48] 2013/09/01 10:24 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (110) [10:28:48] 2013/09/01 10:27 OK z-dat-s4-a MySQL Uptime: 3197019 Threads: 4 Questions: 133949465 Slow queries: 74 Opens: 131 Flush tables: 1 Open tables: 97 Queries per second avg: 41.898 [10:28:48] 2013/09/01 10:27 OK z-dat-s4-a MySQL slave Uptime: 3197019 Threads: 4 Questions: 133949465 Slow queries: 74 Opens: 131 Flush tables: 1 Open tables: 97 Queries per second avg: 41.898 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 314 [10:29:48] 2013/09/01 10:28 OK z-dat-s3-a MySQL Uptime: 410796 Threads: 21 Questions: 379090209 Slow queries: 12972 Opens: 1667014 Flush tables: 1 Open tables: 16384 Queries per second avg: 922.818 [10:29:48] 2013/09/01 10:28 OK z-dat-s3-a MySQL slave Uptime: 410826 Threads: 24 Questions: 379104476 Slow queries: 12973 Opens: 1667066 Flush tables: 1 Open tables: 16384 Queries per second avg: 922.785 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 157 [11:24:51] 2013/09/01 11:23 WARN ortelius Load avg. WARNING - load average: 9.16, 24.99, 15.02 [11:27:52] 2013/09/01 11:26 OK ortelius Load avg. OK - load average: 1.88, 14.55, 12.63 [11:51:54] 2013/09/01 11:44 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.981 second response time [11:54:54] 2013/09/01 11:54 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.243 second response time [12:19:56] 2013/09/01 12:12 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2062.000000 [12:46:57] 2013/09/01 12:40 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2143 [12:47:57] 2013/09/01 12:46 CRIT z-dat-s2-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3631.000000 [13:43:01] 2013/09/01 13:42 OK z-dat-s2-b MySQL slave Uptime: 2399504 Threads: 6 Questions: 2012177910 Slow queries: 22603 Opens: 19627402 Flush tables: 1 Open tables: 257 Queries per second avg: 838.580 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1732 [14:46:03] 2013/09/01 14:44 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3592.000000 [15:08:05] 2013/09/01 15:07 CRIT z-dat-s2-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3610.000000 [15:13:06] 2013/09/01 15:12 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3526.000000 [15:15:06] 2013/09/01 15:08 WARN wolfsbane / DISK WARNING - free space: / 5611 MB (18% inode=93%): [15:15:06] 2013/09/01 15:08 WARN wolfsbane /tmp DISK WARNING - free space: / 5611 MB (18% inode=93%): [15:48:07] 2013/09/01 15:47 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1791.000000 [17:08:12] 2013/09/01 17:07 OK wolfsbane / DISK OK - free space: / 7323 MB (24% inode=93%): [17:08:12] 2013/09/01 17:07 OK wolfsbane /tmp DISK OK - free space: / 7343 MB (24% inode=93%): [22:42:45] nacht ts [23:27:30] 2013/09/01 23:20 CRIT ortelius Load avg. CRITICAL - load average: 47.95, 38.51, 21.14 [23:40:31] 2013/09/01 23:35 WARN wolfsbane Load avg. WARNING - load average: 9.03, 16.04, 13.02 [23:41:31] 2013/09/01 23:40 OK wolfsbane Load avg. OK - load average: 8.99, 14.81, 12.78