[01:01:54] 2014/02/06 01:01 WARN cassia /sql DISK WARNING - free space: /sql 71779 MB (6% inode=99%): [01:02:54] 2014/02/06 01:02 CRIT cassia /sql DISK CRITICAL - free space: /sql 70534 MB (5% inode=99%): [02:00:58] 2014/02/06 02:00 WARN cassia /sql DISK WARNING - free space: /sql 71722 MB (6% inode=99%): [02:01:58] 2014/02/06 02:01 CRIT cassia /sql DISK CRITICAL - free space: /sql 70472 MB (5% inode=99%): [03:01:02] 2014/02/06 03:00 WARN cassia /sql DISK WARNING - free space: /sql 71620 MB (6% inode=99%): [03:02:02] 2014/02/06 03:01 CRIT cassia /sql DISK CRITICAL - free space: /sql 70370 MB (5% inode=99%): [04:39:08] 2014/02/06 04:38 WARN cassia /sql DISK WARNING - free space: /sql 73466 MB (6% inode=99%): [04:40:08] 2014/02/06 04:39 CRIT cassia /sql DISK CRITICAL - free space: /sql 69960 MB (5% inode=99%): [07:04:17] 2014/02/06 07:03 WARN cassia /sql DISK WARNING - free space: /sql 72026 MB (6% inode=99%): [07:13:18] 2014/02/06 07:12 CRIT cassia /sql DISK CRITICAL - free space: /sql 70893 MB (5% inode=99%): [09:02:23] 2014/02/06 09:01 WARN cassia /sql DISK WARNING - free space: /sql 71657 MB (6% inode=99%): [09:03:23] 2014/02/06 09:02 CRIT cassia /sql DISK CRITICAL - free space: /sql 70755 MB (5% inode=99%): [09:10:23] 2014/02/06 09:10 CRIT nightshade / Connection refused or timed out [09:10:23] 2014/02/06 09:09 CRIT nightshade /tmp Connection refused or timed out [09:10:23] 2014/02/06 09:09 CRIT nightshade /var Timeout while attempting connection [09:10:23] 2014/02/06 09:09 CRIT nightshade /var/tmp Timeout while attempting connection [09:10:24] 2014/02/06 09:09 CRIT nightshade Environment IPMI Connection refused or timed out [09:10:24] 2014/02/06 09:09 CRIT nightshade NTP CRITICAL - Socket timeout after 10 seconds [09:10:24] 2014/02/06 09:09 CRIT nightshade PING CRITICAL - Host Unreachable (nightshade) [09:10:25] 2014/02/06 09:10 CRIT nightshade SMTP No route to host [09:10:25] 2014/02/06 09:10 CRIT nightshade SSH No route to host [09:10:26] 2014/02/06 09:09 CRIT nightshade Sensors Timeout while attempting connection [09:14:23] 2014/02/06 09:14 OK nightshade / DISK OK - free space: / 1592 MB (89% inode=94%): [09:14:23] 2014/02/06 09:14 OK nightshade /tmp DISK OK - free space: /tmp 4317 MB (96% inode=99%): [09:14:23] 2014/02/06 09:13 OK nightshade PING PING OK - Packet loss = 0%, RTA = 0.30 ms [09:14:23] 2014/02/06 09:14 OK nightshade SMTP SMTP OK - 0.057 sec. response time [09:14:23] 2014/02/06 09:14 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [09:15:23] 2014/02/06 09:14 OK nightshade /var DISK OK - free space: /var 9365 MB (70% inode=47%): [09:15:23] 2014/02/06 09:14 OK nightshade /var/tmp DISK OK - free space: /var/tmp 872 MB (98% inode=99%): [09:15:23] 2014/02/06 09:14 OK nightshade Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [09:15:23] 2014/02/06 09:14 OK nightshade Load avg. OK - load average: 0.38, 0.16, 0.06 [09:15:23] 2014/02/06 09:14 OK nightshade NTP NTP OK: Offset 0.061102 secs [09:15:24] 2014/02/06 09:14 OK nightshade Sensors sensor ok [09:15:24] 2014/02/06 09:14 ?? nightshade Sun Grid Engine execd Cannot execute /sge/GE/bin/linux-x64/qhost [09:17:23] 2014/02/06 09:16 CRIT nightshade Sun Grid Engine execd CRITICAL: execd not communicating [09:32:24] 2014/02/06 09:31 WARN cassia /sql DISK WARNING - free space: /sql 71409 MB (6% inode=99%): [09:33:24] 2014/02/06 09:32 CRIT cassia /sql DISK CRITICAL - free space: /sql 70577 MB (5% inode=99%): [09:40:25] 2014/02/06 09:39 OK nightshade Sun Grid Engine execd Host and Queues Ok [09:48:26] 2014/02/06 09:47 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.008 second response time [09:55:25] 2014/02/06 09:54 CRIT z-dat-s4-a MySQL slave Slave IO: No Slave SQL: No Seconds Behind Master: (null) [10:15:26] 2014/02/06 10:08 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [10:16:27] 2014/02/06 10:15 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [10:39:27] 2014/02/06 10:32 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [10:48:27] 2014/02/06 10:47 CRIT ortelius Sun Grid Engine execd Connection refused by host [10:49:27] 2014/02/06 10:48 CRIT ortelius /tmp Connection refused or timed out [10:49:28] 2014/02/06 10:49 CRIT ortelius DiskSuite Connection refused or timed out [10:49:28] 2014/02/06 10:49 CRIT ortelius Environment IPMI Connection refused or timed out [10:49:28] 2014/02/06 10:49 CRIT ortelius Load avg. Connection refused or timed out [10:49:28] 2014/02/06 10:48 CRIT ortelius SSH CRITICAL - Socket timeout after 10 seconds [10:50:06] [[Special:Log/newusers]] create 10 * SelinaivaRenneraI * (New user account) [10:50:27] 2014/02/06 10:49 CRIT ortelius / Connection refused or timed out [10:50:27] 2014/02/06 10:49 CRIT ortelius NTP CRITICAL - Socket timeout after 10 seconds [10:50:27] 2014/02/06 10:49 CRIT ortelius PING CRITICAL - Host Unreachable (ortelius) [10:50:27] 2014/02/06 10:49 CRIT ortelius SMTP No route to host [10:51:27] 2014/02/06 10:51 OK ortelius Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [10:51:27] 2014/02/06 10:51 OK ortelius Load avg. OK - load average: 2.15, 0.79, 0.29 [10:51:27] 2014/02/06 10:51 OK ortelius NTP NTP OK: Offset -0.002384 secs [10:52:27] 2014/02/06 10:51 OK ortelius / DISK OK - free space: / 15211 MB (50% inode=91%): [10:52:27] 2014/02/06 10:51 OK ortelius /tmp DISK OK - free space: /tmp 27767 MB (99% inode=99%): [10:52:27] 2014/02/06 10:52 OK ortelius DiskSuite OK - No disk failures detected [10:52:27] 2014/02/06 10:51 OK ortelius PING PING OK - Packet loss = 0%, RTA = 0.55 ms [10:52:27] 2014/02/06 10:51 OK ortelius SMTP SMTP OK - 0.284 sec. response time [10:52:28] 2014/02/06 10:51 OK ortelius SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:52:28] 2014/02/06 10:51 WARN ortelius Sun Grid Engine execd NRPE: Unable to read output [10:52:29] 2014/02/06 10:51 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.048 second response time [10:55:27] 2014/02/06 10:54 WARN cassia /sql DISK WARNING - free space: /sql 71592 MB (6% inode=99%): [10:56:27] 2014/02/06 10:55 CRIT cassia /sql DISK CRITICAL - free space: /sql 70690 MB (5% inode=99%): [11:14:51] [[Special:Log/newusers]] create 10 * Lovella226 * (New user account) [11:16:29] 2014/02/06 11:09 CRIT z-dat-s5-b MySQL slave Slave IO: No Slave SQL: No Seconds Behind Master: (null) [11:21:29] 2014/02/06 11:14 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: Unknown database 'wikidatawiki_p' [11:33:29] 2014/02/06 11:32 WARN cassia /sql DISK WARNING - free space: /sql 71861 MB (6% inode=99%): [11:34:29] 2014/02/06 11:33 CRIT cassia /sql DISK CRITICAL - free space: /sql 70592 MB (5% inode=99%): [12:08:32] 2014/02/06 12:01 CRIT z-dat-s2-b MySQL Can't connect to MySQL server on 'z-dat-s2-b' (111) [12:08:32] 2014/02/06 12:01 CRIT z-dat-s2-b MySQL slave Can't connect to MySQL server on 'z-dat-s2-b' (111) [12:30:32] 2014/02/06 12:29 OK z-dat-s2-b MySQL Uptime: 1719 Threads: 20 Questions: 3015 Slow queries: 1 Opens: 218 Flush tables: 1 Open tables: 210 Queries per second avg: 1.753 [12:30:32] 2014/02/06 12:29 OK z-dat-s2-b MySQL slave Uptime: 1723 Threads: 21 Questions: 3137 Slow queries: 1 Opens: 237 Flush tables: 1 Open tables: 229 Queries per second avg: 1.820 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1713 [12:32:31] 2014/02/06 12:31 WARN cassia /sql DISK WARNING - free space: /sql 71762 MB (6% inode=99%): [12:33:31] 2014/02/06 12:32 CRIT cassia /sql DISK CRITICAL - free space: /sql 70488 MB (5% inode=99%): [12:41:31] 2014/02/06 12:34 WARN ptolemy Load avg. WARNING - load average: 23.16, 20.38, 11.76 [12:44:31] 2014/02/06 12:43 OK ptolemy Load avg. OK - load average: 6.84, 14.77, 11.13 [12:47:32] 2014/02/06 12:40 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [12:49:32] 2014/02/06 12:48 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [13:09:33] 2014/02/06 13:02 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [13:32:35] 2014/02/06 13:31 WARN cassia /sql DISK WARNING - free space: /sql 71655 MB (6% inode=99%): [13:33:35] 2014/02/06 13:32 CRIT cassia /sql DISK CRITICAL - free space: /sql 70423 MB (5% inode=99%): [13:39:35] 2014/02/06 13:38 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.003 second response time [13:53:37] 2014/02/06 13:52 WARN cassia /sql DISK WARNING - free space: /sql 71243 MB (6% inode=99%): [13:54:37] 2014/02/06 13:53 CRIT cassia /sql DISK CRITICAL - free space: /sql 70037 MB (5% inode=99%): [13:56:37] 2014/02/06 13:56 WARN cassia /sql DISK WARNING - free space: /sql 71945 MB (6% inode=99%): [13:57:38] 2014/02/06 13:57 CRIT cassia /sql DISK CRITICAL - free space: /sql 70392 MB (5% inode=99%): [14:31:41] 2014/02/06 14:31 WARN cassia /sql DISK WARNING - free space: /sql 71564 MB (6% inode=99%): [14:32:41] 2014/02/06 14:32 CRIT cassia /sql DISK CRITICAL - free space: /sql 70332 MB (5% inode=99%): [15:35:43] 2014/02/06 15:35 WARN cassia /sql DISK WARNING - free space: /sql 71487 MB (6% inode=99%): [15:36:43] 2014/02/06 15:36 CRIT cassia /sql DISK CRITICAL - free space: /sql 69533 MB (5% inode=99%): [15:38:43] 2014/02/06 15:38 WARN cassia /sql DISK WARNING - free space: /sql 72153 MB (6% inode=99%): [15:39:44] 2014/02/06 15:39 CRIT cassia /sql DISK CRITICAL - free space: /sql 70190 MB (5% inode=99%): [16:40:46] 2014/02/06 16:33 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2073 [17:28:51] 2014/02/06 17:28 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3604 [17:58:52] 2014/02/06 17:58 WARN cassia /sql DISK WARNING - free space: /sql 71331 MB (6% inode=99%): [18:00:52] 2014/02/06 18:00 CRIT cassia /sql DISK CRITICAL - free space: /sql 70946 MB (5% inode=99%): [18:31:54] 2014/02/06 18:31 WARN cassia /sql DISK WARNING - free space: /sql 71750 MB (6% inode=99%): [18:32:54] 2014/02/06 18:32 CRIT cassia /sql DISK CRITICAL - free space: /sql 70910 MB (5% inode=99%): [18:35:15] [[File:Wikimedia Community Logo-Toolserver.png]] ! 10https://wiki.toolserver.org/w/index.php?diff=8337&oldid=8148&rcid=22976 * 95.211.190.207 * (+195) () [18:40:21] [[File:Wikimedia Community Logo-Toolserver.png]] 10https://wiki.toolserver.org/w/index.php?diff=8338&oldid=8337&rcid=22977 * Legoktm * (-195) (rvv) [18:46:56] 2014/02/06 18:40 WARN z-dat-s2-b /sql DISK WARNING - free space: /sql 121440 MB (10% inode=99%): [18:47:25] [[File:Wikimedia Community Logo-Toolserver.png]] ! 10https://wiki.toolserver.org/w/index.php?diff=8339&oldid=8338&rcid=22978 * 95.211.190.207 * (+195) () [19:00:58] 2014/02/06 18:52 CRIT daphne s4 replag (Service Check Timed Out) [19:03:58] 2014/02/06 19:02 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 179.000000 [19:09:35] [[File:Wikimedia Community Logo-Toolserver.png]] ! 10https://wiki.toolserver.org/w/index.php?diff=8340&oldid=8339&rcid=22979 * 195.88.252.132 * (+195) () [19:14:02] [[File:Wikimedia Community Logo-Toolserver.png]] 10https://wiki.toolserver.org/w/index.php?diff=8341&oldid=8340&rcid=22980 * Valhallasw * (-390) (rv) [19:14:19] [[Special:Log/protect]] protect 10 * Valhallasw * (protected "[[02File:Wikimedia Community Logo-Toolserver.png10]]" [edit=autoconfirmed] (indefinite) [move=autoconfirmed] (indefinite) [delete=autoconfirmed] (indefinite) [upload=autoconfirmed] (indefinite): Excessive vandalism) [19:23:00] 2014/02/06 19:14 CRIT daphne s4 replag (Service Check Timed Out) [19:26:01] 2014/02/06 19:25 OK z-dat-s1-b MySQL Uptime: 58 Threads: 2 Questions: 494 Slow queries: 0 Opens: 790 Flush tables: 1 Open tables: 204 Queries per second avg: 8.517 [19:26:01] 2014/02/06 19:25 WARN z-dat-s1-b MySQL slave No slaves defined [19:32:01] 2014/02/06 19:31 WARN cassia /sql DISK WARNING - free space: /sql 71987 MB (6% inode=99%): [19:33:01] 2014/02/06 19:32 CRIT cassia /sql DISK CRITICAL - free space: /sql 70726 MB (5% inode=99%): [19:38:02] 2014/02/06 19:37 CRIT willow / Connection refused or timed out [19:38:02] 2014/02/06 19:37 CRIT willow /tmp Connection refused or timed out [19:38:02] 2014/02/06 19:36 CRIT willow Environment IPMI Connection refused or timed out [19:38:02] 2014/02/06 19:36 CRIT willow Load avg. Connection refused or timed out [19:38:02] 2014/02/06 19:37 CRIT willow NTP CRITICAL - Socket timeout after 10 seconds [19:38:03] 2014/02/06 19:37 CRIT willow PING CRITICAL - Host Unreachable (willow) [19:38:03] 2014/02/06 19:37 CRIT willow SMTP CRITICAL - Socket timeout after 10 seconds [19:38:04] 2014/02/06 19:37 CRIT willow SSH No route to host [19:38:04] 2014/02/06 19:37 CRIT willow Sun Grid Engine execd Connection refused or timed out [19:39:02] 2014/02/06 19:37 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 680.000000 [19:40:03] 2014/02/06 19:39 OK willow / DISK OK - free space: / 23055 MB (22% inode=99%): [19:40:03] 2014/02/06 19:39 OK willow /tmp DISK OK - free space: / 23055 MB (22% inode=99%): [19:40:03] 2014/02/06 19:39 OK willow Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [19:41:02] 2014/02/06 19:39 OK willow Load avg. OK - load average: 2.86, 0.96, 0.35 [19:41:02] 2014/02/06 19:40 OK willow NTP NTP OK: Offset 0.235007 secs [19:41:02] 2014/02/06 19:40 OK willow PING PING OK - Packet loss = 0%, RTA = 0.26 ms [19:41:02] 2014/02/06 19:40 OK willow SMTP SMTP OK - 0.110 sec. response time [19:41:02] 2014/02/06 19:40 OK willow SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:41:03] 2014/02/06 19:40 WARN willow Sun Grid Engine execd NRPE: Unable to read output [20:01:03] 2014/02/06 20:00 WARN cassia /sql DISK WARNING - free space: /sql 71978 MB (6% inode=99%): [20:02:03] 2014/02/06 20:01 CRIT cassia /sql DISK CRITICAL - free space: /sql 70705 MB (5% inode=99%): [20:07:03] 2014/02/06 20:06 OK z-dat-s2-b /sql DISK OK - free space: /sql 140819 MB (12% inode=99%): [20:10:31] Why aren't reboots announced here? [20:14:24] Dispenser: it was more of an emergency case [20:14:32] Dispenser: did you get my memo? [21:06:08] Marlen Caemmerer * Re: [Toolserver-l] Do the problems persist? [21:36:07] 2014/02/06 21:35 CRIT z-dat-s1-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 904594 [22:19:12] 2014/02/06 22:18 WARN cassia /sql DISK WARNING - free space: /sql 71006 MB (6% inode=99%): [22:20:12] 2014/02/06 22:19 CRIT cassia /sql DISK CRITICAL - free space: /sql 70492 MB (5% inode=99%): [22:33:46] Betacommand: It depends if I'm allocated 24 TB for the important I want to make. If not then I declare there isn't a future with labs. [22:40:14] 2014/02/06 22:39 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3586 [22:42:13] 2014/02/06 22:41 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3603 [22:46:13] 2014/02/06 22:45 WARN cassia /sql DISK WARNING - free space: /sql 71259 MB (6% inode=99%): [22:47:13] 2014/02/06 22:46 CRIT cassia /sql DISK CRITICAL - free space: /sql 70419 MB (5% inode=99%): [23:08:13] 2014/02/06 23:07 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3491 [23:22:15] 2014/02/06 23:21 WARN cassia /sql DISK WARNING - free space: /sql 72340 MB (6% inode=99%): [23:23:15] 2014/02/06 23:22 CRIT cassia /sql DISK CRITICAL - free space: /sql 70339 MB (5% inode=99%): [23:25:15] 2014/02/06 23:24 OK z-dat-s2-b MySQL slave Uptime: 41023 Threads: 33 Questions: 5283803 Slow queries: 1025 Opens: 138788 Flush tables: 1 Open tables: 256 Queries per second avg: 128.800 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1775 [23:39:17] 2014/02/06 23:32 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2080 [23:54:17] 2014/02/06 23:53 OK z-dat-s2-b MySQL slave Uptime: 42763 Threads: 21 Questions: 5616294 Slow queries: 1115 Opens: 143785 Flush tables: 1 Open tables: 256 Queries per second avg: 131.335 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1736