[00:02:19] 2013/06/10 00:02 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3534 [00:08:19] 2013/06/10 00:08 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3601 [00:09:19] 2013/06/10 00:09 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3597 [00:10:20] 2013/06/10 00:10 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3609 [00:11:19] 2013/06/10 00:11 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3573 [00:18:20] 2013/06/10 00:18 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3630 [00:35:20] 2013/06/10 00:35 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3292 [00:46:21] 2013/06/10 00:46 OK z-dat-s2-b MySQL slave Uptime: 1873370 Threads: 7 Questions: 2091205138 Slow queries: 41338 Opens: 12959327 Flush tables: 1 Open tables: 256 Queries per second avg: 1116.279 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1608 [01:17:29] [[Special:Log/newusers]] create 10 * Runis * (New user account) [03:08:29] 2013/06/10 03:07 OK wolfsbane / DISK OK - free space: / 7326 MB (24% inode=92%): [03:08:29] 2013/06/10 03:08 OK wolfsbane /tmp DISK OK - free space: / 7327 MB (24% inode=92%): [06:51:19] [[Special:Log/newusers]] create 10 * Emaniuz * (New user account) [07:29:40] 2013/06/10 07:28 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.969 second response time [07:31:40] 2013/06/10 07:31 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.270 second response time [09:30:52] 2013/06/10 09:24 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2174 [09:30:52] 2013/06/10 09:24 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2185.000000 [09:54:54] 2013/06/10 09:54 CRIT daphne s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3602.000000 [09:55:54] 2013/06/10 09:55 CRIT daphne MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3644 [10:15:54] 2013/06/10 10:15 OK nightshade aliasd TCP OK - 0.007 second response time on port 984 [500 Not found.] [10:16:54] 2013/06/10 10:16 OK yarrow aliasd TCP OK - 0.009 second response time on port 984 [500 Not found.] [11:23:00] 2013/06/10 11:15 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2183 [11:25:00] 2013/06/10 11:21 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2001.000000 [11:27:00] 2013/06/10 11:19 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2165 [11:27:00] 2013/06/10 11:20 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2187 [11:28:00] 2013/06/10 11:26 CRIT z-dat-s7-a wikidata replag (Service Check Timed Out) [11:29:01] 2013/06/10 11:27 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [11:29:01] 2013/06/10 11:28 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (111) [11:34:02] 2013/06/10 11:27 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1813 [11:35:01] 2013/06/10 11:28 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (111) [11:36:01] 2013/06/10 11:28 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [11:37:02] 2013/06/10 11:35 OK z-dat-s4-a MySQL slave Uptime: 1133948 Threads: 1 Questions: 80157971 Slow queries: 52 Opens: 267 Flush tables: 1 Open tables: 259 Queries per second avg: 70.689 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1774 [11:47:04] 2013/06/10 11:46 OK z-dat-s7-a MySQL Uptime: 1085 Threads: 3 Questions: 217 Slow queries: 0 Opens: 88 Flush tables: 1 Open tables: 77 Queries per second avg: 0.200 [11:47:04] 2013/06/10 11:46 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3393 [11:47:04] 2013/06/10 11:46 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3320.000000 [11:52:04] 2013/06/10 11:51 CRIT z-dat-s7-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3621.000000 [11:53:04] 2013/06/10 11:51 OK z-dat-s6-a MySQL slave Uptime: 1818434 Threads: 6 Questions: 1838458810 Slow queries: 109196 Opens: 3131167 Flush tables: 1 Open tables: 3409 Queries per second avg: 1011.12 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1429 [11:53:04] 2013/06/10 11:52 OK z-dat-s7-a MySQL slave Uptime: 1451 Threads: 4 Questions: 178242 Slow queries: 24 Opens: 537 Flush tables: 1 Open tables: 515 Queries per second avg: 122.840 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1744 [11:54:04] 2013/06/10 11:52 OK z-dat-s3-a MySQL Uptime: 1485 Threads: 1 Questions: 13 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 15 Queries per second avg: 0.8 [11:54:04] 2013/06/10 11:52 OK z-dat-s3-a MySQL slave Uptime: 1493 Threads: 12 Questions: 831 Slow queries: 0 Opens: 132 Flush tables: 1 Open tables: 78 Queries per second avg: 0.556 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:01:05] 2013/06/10 11:53 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4435 [12:27:06] 2013/06/10 12:25 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3525 [12:39:07] 2013/06/10 12:37 OK z-dat-s3-a MySQL slave Uptime: 4193 Threads: 17 Questions: 3072967 Slow queries: 252 Opens: 22166 Flush tables: 1 Open tables: 13626 Queries per second avg: 732.880 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1732 [12:45:07] 2013/06/10 12:44 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3494 [12:45:07] 2013/06/10 12:44 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3418.000000 [12:51:08] 2013/06/10 12:44 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1942 [13:09:10] 2013/06/10 13:09 OK daphne MySQL slave Uptime: 4555352 Threads: 7 Questions: 464853276 Slow queries: 902225 Opens: 140445 Flush tables: 1 Open tables: 2066 Queries per second avg: 102.45 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1592 [13:10:10] 2013/06/10 13:09 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1559.000000 [13:23:10] 2013/06/10 13:22 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3522.000000 [13:24:10] 2013/06/10 13:24 OK z-dat-s2-b MySQL slave Uptime: 1918850 Threads: 8 Questions: 2137144583 Slow queries: 42657 Opens: 13323721 Flush tables: 1 Open tables: 256 Queries per second avg: 1113.763 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1762 [13:39:11] 2013/06/10 13:38 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1684.000000 [14:22:01] Hello all [14:49:12] 2013/06/10 14:45 CRIT z-dat-s7-a wikidata replag QUERY CRITICAL: Can't connect to MySQL server on 'z-dat-s7-a' (111) [14:52:13] 2013/06/10 14:45 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (111) [14:52:13] 2013/06/10 14:45 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (111) [14:54:13] 2013/06/10 14:47 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (111) [14:54:13] 2013/06/10 14:47 CRIT z-dat-s6-a MySQL slave Can't connect to MySQL server on 'z-dat-s6-a' (111) [15:02:13] 2013/06/10 15:01 OK z-dat-s6-a MySQL Uptime: 843 Threads: 1 Questions: 33 Slow queries: 0 Opens: 26 Flush tables: 1 Open tables: 17 Queries per second avg: 0.39 [15:02:14] 2013/06/10 15:01 OK z-dat-s6-a MySQL slave Uptime: 850 Threads: 6 Questions: 301 Slow queries: 0 Opens: 52 Flush tables: 1 Open tables: 39 Queries per second avg: 0.354 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1255 [15:05:14] 2013/06/10 14:57 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1878 [15:06:00] I use a script to facilitate newpage patrol (it was written for me; I don't know how it works). It apparently relies on Toolserv, and thus hasn't been working lately -- but my other toolserv-based pages are working fine. [15:06:07] Could anyone take a look, maybe figure out what's wrong? [15:08:42] Dragonfly6-7: what language is it in? [15:09:14] 2013/06/10 15:07 OK z-dat-s3-a MySQL slave Uptime: 13192 Threads: 30 Questions: 16635147 Slow queries: 4486 Opens: 61663 Flush tables: 1 Open tables: 16384 Queries per second avg: 1261.2 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1758 [15:10:14] 2013/06/10 15:09 OK z-dat-s7-a MySQL Uptime: 1478 Threads: 3 Questions: 726 Slow queries: 1 Opens: 98 Flush tables: 1 Open tables: 89 Queries per second avg: 0.491 [15:10:14] 2013/06/10 15:09 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2223 [15:10:14] 2013/06/10 15:09 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1920.000000 [15:15:14] 2013/06/10 15:14 OK z-dat-s7-a MySQL slave Uptime: 1786 Threads: 5 Questions: 103026 Slow queries: 9 Opens: 547 Flush tables: 1 Open tables: 506 Queries per second avg: 57.685 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1691 [15:38:14] 2013/06/10 15:31 WARN wolfsbane / DISK WARNING - free space: / 6260 MB (20% inode=92%): [15:38:14] 2013/06/10 15:32 WARN wolfsbane /tmp DISK WARNING - free space: / 6279 MB (20% inode=92%): [15:39:15] 2013/06/10 15:38 CRIT z-dat-s7-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3659.000000 [15:56:15] 2013/06/10 15:55 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3594.000000 [16:37:20] 2013/06/10 16:36 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1784.000000 [17:43:23] DaBPunkt: Do you know if you recently did a cleanup of the osm tile dir? [18:11:26] 2013/06/10 18:10 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.069 second response time [18:12:26] 2013/06/10 18:12 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.738 second response time [18:13:26] 2013/06/10 18:13 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.047 second response time [18:53:28] 2013/06/10 18:52 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.877 second response time [18:54:28] 2013/06/10 18:54 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.009 second response time [19:08:28] re [19:08:34] apmon: not in the last months [19:24:30] 2013/06/10 19:18 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.150 second response time [19:26:31] 2013/06/10 19:26 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.728 second response time [19:27:31] 2013/06/10 19:27 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.674 second response time [19:29:32] 2013/06/10 19:29 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.924 second response time [19:31:31] 2013/06/10 19:31 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.212 second response time [20:06:32] 2013/06/10 20:02 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 18.440 second response time [20:09:33] 2013/06/10 20:08 CRIT web.amaranth fisheye.toolserver.org HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.834 second response time [20:11:33] 2013/06/10 20:10 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 18.841 second response time [20:14:33] 2013/06/10 20:13 CRIT web.amaranth fisheye.toolserver.org CRITICAL - Socket timeout after 21 seconds [20:16:34] 2013/06/10 20:15 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 18.212 second response time [20:17:34] 2013/06/10 20:16 OK web.amaranth fisheye.toolserver.org HTTP OK: HTTP/1.1 200 OK - 273 bytes in 11.211 second response time [20:23:34] 2013/06/10 20:16 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2158 [20:24:34] 2013/06/10 20:17 CRIT nightshade aliasd Connection refused [20:25:34] 2013/06/10 20:21 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2005.000000 [20:27:34] 2013/06/10 20:20 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2184 [20:27:34] 2013/06/10 20:20 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2191 [20:41:13] DaBPunkt: OK, thanks. The tile server seems to be getting stuck on rendering old low zoom tiles. I'll need to figure out why it is doing that, as it is stalling the requests it should be rendering [20:45:36] 2013/06/10 20:38 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2098 [20:48:36] 2013/06/10 20:47 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3629 [20:51:41] 2013/06/10 20:50 CRIT z-dat-s7-a MySQL slave (Service Check Timed Out) [20:51:41] 2013/06/10 20:49 CRIT z-dat-s7-a wikidata replag (Service Check Timed Out) [20:52:36] 2013/06/10 20:50 CRIT z-dat-s6-a MySQL slave (Service Check Timed Out) [20:57:37] 2013/06/10 20:50 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [20:57:37] 2013/06/10 20:50 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (111) [20:59:38] 2013/06/10 20:59 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.866 second response time [21:00:38] 2013/06/10 21:00 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.200 second response time [21:09:38] 2013/06/10 21:08 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3582 [21:11:38] 2013/06/10 21:10 OK z-dat-s4-a MySQL slave Uptime: 1168449 Threads: 2 Questions: 81604184 Slow queries: 54 Opens: 267 Flush tables: 1 Open tables: 259 Queries per second avg: 69.839 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1484 [21:15:39] 2013/06/10 21:14 OK z-dat-s7-a MySQL Uptime: 1158 Threads: 4 Questions: 3133 Slow queries: 0 Opens: 166 Flush tables: 1 Open tables: 154 Queries per second avg: 2.705 [21:16:39] 2013/06/10 21:15 OK z-dat-s6-a MySQL slave Uptime: 23304 Threads: 10 Questions: 15587363 Slow queries: 3298 Opens: 1427 Flush tables: 1 Open tables: 607 Queries per second avg: 668.870 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1445 [21:23:40] 2013/06/10 21:22 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3580 [21:26:39] 2013/06/10 21:25 OK z-dat-s3-a MySQL Uptime: 1847 Threads: 30 Questions: 12950 Slow queries: 2 Opens: 277 Flush tables: 1 Open tables: 237 Queries per second avg: 7.11 [21:30:40] 2013/06/10 21:29 OK z-dat-s7-a MySQL slave Uptime: 2068 Threads: 7 Questions: 612436 Slow queries: 31 Opens: 623 Flush tables: 1 Open tables: 598 Queries per second avg: 296.148 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1747 [21:31:40] 2013/06/10 21:25 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.289 second response time [21:32:40] 2013/06/10 21:25 CRIT ortelius Load avg. CRITICAL - load average: 181.75, 172.83, 88.06 [21:33:40] 2013/06/10 21:33 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.160 second response time [21:45:51] nacht ts [22:00:42] 2013/06/10 21:59 WARN ortelius Load avg. WARNING - load average: 2.28, 3.76, 19.76 [22:06:43] 2013/06/10 22:05 OK ortelius Load avg. OK - load average: 3.27, 3.36, 14.35 [22:21:43] 2013/06/10 22:20 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3589 [22:29:44] 2013/06/10 22:28 OK z-dat-s3-a MySQL slave Uptime: 5628 Threads: 17 Questions: 6509960 Slow queries: 391 Opens: 22488 Flush tables: 1 Open tables: 13621 Queries per second avg: 1156.709 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1784 [22:39:45] 2013/06/10 22:39 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3484.000000 [23:06:45] 2013/06/10 23:06 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1783.000000 [23:46:47] 2013/06/10 23:46 OK nightshade aliasd TCP OK - 0.004 second response time on port 984 [500 Not found.]