[01:47:42] 2013/07/15 01:41 WARN hemlock / DISK WARNING - free space: / 4192 MB (20% inode=87%): [01:48:42] 2013/07/15 01:41 WARN hemlock /tmp DISK WARNING - free space: / 4192 MB (20% inode=87%): [03:10:47] 2013/07/15 03:10 OK hemlock / DISK OK - free space: / 4220 MB (21% inode=87%): [03:10:48] 2013/07/15 03:10 OK hemlock /tmp DISK OK - free space: / 4220 MB (21% inode=87%): [06:33:59] 2013/07/15 06:27 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2170.000000 [07:17:59] 2013/07/15 07:17 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1730.000000 [07:30:58] 2013/07/15 07:24 WARN hemlock / DISK WARNING - free space: / 4192 MB (20% inode=87%): [07:30:59] 2013/07/15 07:24 WARN hemlock /tmp DISK WARNING - free space: / 4192 MB (20% inode=87%): [08:58:04] Johannes Kroll * [Toolserver-l] user-store not mounted [09:18:04] Marlen Caemmerer * Re: [Toolserver-l] user-store not mounted [09:24:41] HBC bots running on toolserver are logged out. [09:24:48] Is it possible to log them in. [09:24:51] *? [09:29:05] 2013/07/15 09:22 WARN z-dat-s1-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2171 [09:41:04] Johannes Kroll * Re: [Toolserver-l] user-store not mounted [09:48:05] 2013/07/15 09:47 OK z-dat-s1-b MySQL slave Uptime: 4929793 Threads: 8 Questions: 4806767061 Slow queries: 78512 Opens: 25245 Flush tables: 1 Open tables: 782 Queries per second avg: 975.44 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1736 [11:20:08] 2013/07/15 11:13 WARN wolfsbane / DISK WARNING - free space: / 5738 MB (19% inode=93%): [11:21:08] 2013/07/15 11:14 WARN wolfsbane /tmp DISK WARNING - free space: / 5724 MB (19% inode=93%): [11:44:09] 2013/07/15 11:35 CRIT z-dat-s6-a wikidata replag (Service Check Timed Out) [11:48:10] 2013/07/15 11:39 CRIT z-dat-s7-a wikidata replag QUERY CRITICAL: Can't connect to MySQL server on 'z-dat-s7-a' (111) [11:49:10] 2013/07/15 11:48 OK z-dat-s6-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1255.000000 [11:52:10] 2013/07/15 11:45 CRIT yarrow aliasd Connection refused [11:52:10] 2013/07/15 11:47 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [11:52:10] 2013/07/15 11:47 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [11:54:10] 2013/07/15 11:47 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (111) [11:54:10] 2013/07/15 11:47 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (111) [12:01:11] 2013/07/15 11:57 CRIT z-dat-s4-a MySQL Can't connect to MySQL server on 'z-dat-s4-a' (111) [12:01:11] 2013/07/15 11:57 CRIT z-dat-s4-a MySQL slave Can't connect to MySQL server on 'z-dat-s4-a' (111) [12:02:11] 2013/07/15 12:01 OK z-dat-s4-a MySQL Uptime: 264 Threads: 3 Questions: 54 Slow queries: 0 Opens: 25 Flush tables: 1 Open tables: 15 Queries per second avg: 0.204 [12:02:11] 2013/07/15 12:01 OK z-dat-s4-a MySQL slave Uptime: 264 Threads: 3 Questions: 54 Slow queries: 0 Opens: 25 Flush tables: 1 Open tables: 15 Queries per second avg: 0.204 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1359 [12:05:11] 2013/07/15 11:57 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2154 [12:05:11] 2013/07/15 11:58 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2140.000000 [12:08:11] 2013/07/15 12:07 OK wolfsbane / DISK OK - free space: / 6940 MB (23% inode=93%): [12:08:11] 2013/07/15 12:07 OK wolfsbane /tmp DISK OK - free space: / 6941 MB (23% inode=93%): [12:08:11] 2013/07/15 12:07 OK z-dat-s7-a MySQL Uptime: 1197 Threads: 6 Questions: 2745 Slow queries: 2 Opens: 203 Flush tables: 1 Open tables: 189 Queries per second avg: 2.293 [12:08:11] 2013/07/15 12:07 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2303 [12:08:11] 2013/07/15 12:07 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2324.000000 [12:11:10] 2013/07/15 12:09 OK z-dat-s3-a MySQL Uptime: 1346 Threads: 23 Questions: 4714 Slow queries: 0 Opens: 191 Flush tables: 1 Open tables: 149 Queries per second avg: 3.502 [12:11:11] 2013/07/15 12:09 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2559 [12:11:11] 2013/07/15 12:10 OK z-dat-s7-a MySQL slave Uptime: 1388 Threads: 5 Questions: 60836 Slow queries: 4 Opens: 451 Flush tables: 1 Open tables: 434 Queries per second avg: 43.829 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1592 [12:26:10] 2013/07/15 12:21 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.374 second response time [12:27:10] 2013/07/15 12:26 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.438 second response time [12:30:10] 2013/07/15 12:29 CRIT z-dat-s7-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3644.000000 [12:33:11] 2013/07/15 12:31 OK z-dat-s6-a MySQL slave Uptime: 271971 Threads: 6 Questions: 193324004 Slow queries: 18655 Opens: 31513 Flush tables: 1 Open tables: 3125 Queries per second avg: 710.825 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1584 [12:42:12] 2013/07/15 12:41 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3589.000000 [12:46:13] 2013/07/15 12:45 OK z-dat-s6-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1783.000000 [12:49:13] 2013/07/15 12:47 OK z-dat-s3-a MySQL slave Uptime: 3627 Threads: 25 Questions: 2893490 Slow queries: 129 Opens: 12058 Flush tables: 1 Open tables: 11953 Queries per second avg: 797.763 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1651 [13:25:16] 2013/07/15 13:18 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2152.000000 [13:29:16] 2013/07/15 13:22 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2198 [13:34:16] 2013/07/15 13:33 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1793.000000 [13:58:15] 2013/07/15 13:57 CRIT z-dat-s2-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3603.000000 [14:01:15] 2013/07/15 14:00 OK z-dat-s2-b MySQL slave Uptime: 4945055 Threads: 11 Questions: 4651547957 Slow queries: 86895 Opens: 36454186 Flush tables: 1 Open tables: 256 Queries per second avg: 940.646 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1790 [14:25:16] 2013/07/15 14:24 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3594.000000 [17:53:30] 2013/07/15 17:52 WARN web.amaranth fisheye.toolserver.org HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 19.356 second response time [17:56:30] 2013/07/15 17:55 OK web.amaranth fisheye.toolserver.org HTTP OK: HTTP/1.1 200 OK - 273 bytes in 12.058 second response time [18:24:33] 2013/07/15 18:23 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1791.000000 [18:30:33] 2013/07/15 18:29 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.535 second response time [18:31:33] 2013/07/15 18:30 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [18:33:26] 2013/07/15 18:31 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.278 second response time [19:57:38] 2013/07/15 19:50 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [20:20:39] 2013/07/15 20:19 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.906 second response time [20:21:38] 2013/07/15 20:20 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [20:24:39] 2013/07/15 20:23 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.590 second response time [20:25:40] 2013/07/15 20:24 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.922 second response time [20:29:40] 2013/07/15 20:28 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.241 second response time [21:58:44] 2013/07/15 21:51 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2046.000000 [23:30:48] 2013/07/15 23:29 CRIT z-dat-s2-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3603.000000