[00:09:16] Hi Danny_B [03:08:30] 2013/09/29 03:02 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2093 [03:08:30] 2013/09/29 03:02 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2093.000000 [03:17:31] 2013/09/29 03:17 OK z-dat-s5-b MySQL slave Uptime: 1697945 Threads: 5 Questions: 2723380917 Slow queries: 8582 Opens: 325889 Flush tables: 1 Open tables: 255 Queries per second avg: 1603.927 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1736 [03:17:31] 2013/09/29 03:17 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1738.000000 [03:52:33] 2013/09/29 03:51 WARN ptolemy /sql DISK WARNING - free space: /sql 90264 MB (11% inode=99%): [03:53:33] 2013/09/29 03:52 CRIT ptolemy /sql DISK CRITICAL - free space: /sql 87486 MB (10% inode=99%): [06:29:17] Dispenser: around now? [06:29:24] yes [06:29:59] Dispenser: you have >17gb on user store, do you need everything? [06:30:31] Its temporary, I can delete it now [06:30:37] awesome [06:30:41] please do so [06:30:51] usr store is running out of space [06:31:20] pls consider enhancing your scripts with deletion of tmp stuff after finished, thanks [06:31:21] I'm working on https://commons.wikimedia.org/wiki/User:Dispenser/Absurd_overhead [06:32:56] interesting! [06:33:29] I was working on extending it to PNG [06:34:44] 11 GB gone [06:34:51] \o/ [06:34:55] thanks [06:57:30] Dispenser: the file size column doesn't sort properly, you may like to consider removing of "KB" and thousand separator [06:57:57] Can't we just fix MediaWiki? :-P [06:59:01] I'm actually a bit more concerned about McZusatz march to eliminate metadata [06:59:53] up to you... i was just suggesting to make it more usable [09:01:53] 2013/09/29 09:01 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:06:54] 2013/09/29 09:06 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 59521 MB (9% inode=99%): [09:10:54] 2013/09/29 09:10 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:55] 2013/09/29 09:11 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 59515 MB (9% inode=99%): [09:17:56] 2013/09/29 09:11 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [09:17:56] 2013/09/29 09:11 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [09:29:57] 2013/09/29 09:29 OK z-dat-s3-a MySQL Uptime: 1093 Threads: 5 Questions: 92 Slow queries: 0 Opens: 46 Flush tables: 1 Open tables: 38 Queries per second avg: 0.84 [09:29:57] 2013/09/29 09:29 OK z-dat-s3-a MySQL slave Uptime: 1103 Threads: 11 Questions: 231 Slow queries: 0 Opens: 115 Flush tables: 1 Open tables: 106 Queries per second avg: 0.209 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1111 [12:04:06] 2013/09/29 11:57 WARN ortelius Load avg. WARNING - load average: 19.81, 16.67, 9.01 [12:10:06] 2013/09/29 12:09 CRIT ortelius Load avg. CRITICAL - load average: 37.09, 25.29, 15.00 [12:17:06] 2013/09/29 12:16 WARN ortelius Load avg. WARNING - load average: 18.32, 23.54, 18.45 [12:45:06] 2013/09/29 12:44 CRIT ortelius Load avg. CRITICAL - load average: 28.70, 24.25, 20.42 [12:47:06] 2013/09/29 12:40 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1964.000000 [12:57:06] 2013/09/29 12:50 WARN wolfsbane / DISK WARNING - free space: / 6220 MB (20% inode=93%): [12:57:06] 2013/09/29 12:50 WARN wolfsbane /tmp DISK WARNING - free space: / 6230 MB (20% inode=93%): [13:11:13] Dispenser: youtube folder - needed? [13:11:35] Yes [13:12:15] Are we really that short on space? [13:18:07] 2013/09/29 13:15 WARN wolfsbane Load avg. WARNING - load average: 26.61, 22.76, 16.50 [13:28:07] 2013/09/29 13:27 CRIT wolfsbane Load avg. CRITICAL - load average: 34.63, 27.22, 20.80 [13:39:08] 2013/09/29 13:37 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [13:41:08] 2013/09/29 13:40 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 59159 MB (9% inode=99%): [13:41:08] 2013/09/29 13:37 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [13:42:08] 2013/09/29 13:40 OK z-dat-s4-a MySQL slave Uptime: 2172975 Threads: 3 Questions: 159179779 Slow queries: 99 Opens: 319 Flush tables: 1 Open tables: 250 Queries per second avg: 73.254 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [13:45:08] 2013/09/29 13:44 WARN wolfsbane Load avg. WARNING - load average: 7.04, 13.64, 19.61 [13:49:08] 2013/09/29 13:48 WARN ortelius Load avg. WARNING - load average: 1.86, 13.64, 19.49 [13:50:08] 2013/09/29 13:49 OK wolfsbane Load avg. OK - load average: 1.74, 6.25, 14.66 [13:54:08] 2013/09/29 13:53 OK ortelius Load avg. OK - load average: 1.30, 5.99, 14.44 [13:55:48] henna was owned [13:55:49] today [13:55:53] check his sites [13:56:01] fuzzybeans.net [13:56:14] /w in 12 [14:33:11] 2013/09/29 14:32 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1793.000000 [14:38:11] 2013/09/29 14:37 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [14:42:11] 2013/09/29 14:41 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 59994 MB (9% inode=99%): [14:55:12] 2013/09/29 14:49 CRIT wolfsbane Load avg. CRITICAL - load average: 68.03, 126.07, 65.84 [14:56:12] 2013/09/29 14:54 WARN ortelius Load avg. WARNING - load average: 6.07, 18.92, 12.76 [14:58:13] 2013/09/29 14:57 OK ortelius Load avg. OK - load average: 1.79, 13.11, 11.33 [15:13:12] 2013/09/29 15:06 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1958.000000 [15:15:12] 2013/09/29 15:14 WARN wolfsbane Load avg. WARNING - load average: 2.91, 5.23, 19.72 [15:21:12] 2013/09/29 15:20 OK wolfsbane Load avg. OK - load average: 3.04, 3.94, 14.37 [15:54:16] 2013/09/29 15:53 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1754.000000 [16:37:17] 2013/09/29 16:33 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [16:39:16] Danny_B: I've compressed them from 517 MB to 208 MB [16:39:17] 2013/09/29 16:38 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.003 second response time [16:43:17] 2013/09/29 16:42 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [16:45:17] 2013/09/29 16:44 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 59769 MB (9% inode=99%): [18:26:25] 2013/09/29 18:25 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [18:27:25] 2013/09/29 18:26 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 59579 MB (9% inode=99%): [18:52:28] 2013/09/29 18:51 CRIT wolfsbane / DISK CRITICAL - free space: / 3281 MB (10% inode=93%): [18:53:28] 2013/09/29 18:52 WARN wolfsbane / DISK WARNING - free space: / 3299 MB (11% inode=93%): [18:55:28] 2013/09/29 18:54 CRIT wolfsbane / DISK CRITICAL - free space: / 3286 MB (10% inode=93%): [18:55:28] 2013/09/29 18:54 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3286 MB (10% inode=93%): [21:56:02] CREATE TABLE IF NOT EXISTS is broken for me :-(. Inserts rows despite that the table already exists