[08:12:11] <wikirc>	 [[Special:Log/newusers]] create 10 * Fredric867 *  (New user account)
[08:53:31] <tsnag>	 2014/02/04 08:53 CRIT nightshade Environment IPMI Timeout while attempting connection
[08:53:31] <tsnag>	 2014/02/04 08:53 CRIT nightshade Sensors Timeout while attempting connection
[08:54:31] <tsnag>	 2014/02/04 08:53 CRIT nightshade / Timeout while attempting connection
[08:54:31] <tsnag>	 2014/02/04 08:53 CRIT nightshade /tmp Timeout while attempting connection
[08:54:31] <tsnag>	 2014/02/04 08:53 CRIT nightshade /var Timeout while attempting connection
[08:54:31] <tsnag>	 2014/02/04 08:53 CRIT nightshade /var/tmp Timeout while attempting connection
[08:54:31] <tsnag>	 2014/02/04 08:53 CRIT nightshade NTP CRITICAL - Socket timeout after 10 seconds
[08:54:31] <tsnag>	 2014/02/04 08:53 CRIT nightshade PING PING CRITICAL - Packet loss = 100%
[08:54:32] <tsnag>	 2014/02/04 08:53 CRIT nightshade SMTP CRITICAL - Socket timeout after 10 seconds
[08:54:32] <tsnag>	 2014/02/04 08:53 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds
[08:56:31] <tsnag>	 2014/02/04 08:55  OK  nightshade / DISK OK - free space: / 1592 MB (89% inode=94%):
[08:56:31] <tsnag>	 2014/02/04 08:55  OK  nightshade /tmp DISK OK - free space: /tmp 4317 MB (96% inode=99%):
[08:56:31] <tsnag>	 2014/02/04 08:55  OK  nightshade /var DISK OK - free space: /var 9388 MB (70% inode=47%):
[08:56:31] <tsnag>	 2014/02/04 08:55  OK  nightshade /var/tmp DISK OK - free space: /var/tmp 872 MB (98% inode=99%):
[08:56:31] <tsnag>	 2014/02/04 08:56  OK  nightshade Environment IPMI ok:  temperature ok fan ok voltage ok chassis ok
[08:56:31] <tsnag>	 2014/02/04 08:55  OK  nightshade Load avg. OK - load average: 0.84, 0.19, 0.06
[08:56:31] <tsnag>	 2014/02/04 08:55  OK  nightshade PING PING OK - Packet loss = 0%, RTA = 0.22 ms
[08:56:32] <tsnag>	 2014/02/04 08:55  OK  nightshade SMTP SMTP OK - 0.119 sec. response time
[08:56:32] <tsnag>	 2014/02/04 08:55  OK  nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0)
[08:56:33] <tsnag>	 2014/02/04 08:56  OK  nightshade Sensors sensor ok
[08:56:33] <tsnag>	 2014/02/04 08:55  ??  nightshade Sun Grid Engine execd Cannot execute /sge/GE/bin/linux-x64/qhost
[08:56:34] <tsnag>	 2014/02/04 08:56  OK  nightshade aliasd TCP OK - 0.001 second response time on port 984 [500 Not found.]
[08:57:31] <tsnag>	 2014/02/04 08:56 CRIT nightshade Sun Grid Engine execd CRITICAL: execd not communicating
[09:04:32] <tsnag>	 2014/02/04 09:03 CRIT cassia /sql DISK CRITICAL - free space: /sql 70991 MB (5% inode=99%):
[09:32:32] <tsnag>	 2014/02/04 09:31 WARN cassia /sql DISK WARNING - free space: /sql 71597 MB (6% inode=99%):
[09:33:32] <tsnag>	 2014/02/04 09:32 CRIT cassia /sql DISK CRITICAL - free space: /sql 70800 MB (5% inode=99%):
[10:08:33] <tsnag>	 2014/02/04 10:07 CRIT z-dat-s7-a /sql CHECK_NRPE: Socket timeout after 30 seconds.
[10:09:33] <tsnag>	 2014/02/04 10:08 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out)
[10:10:34] <tsnag>	 2014/02/04 10:07 CRIT z-dat-s7-a SSH CRITICAL - Socket timeout after 10 seconds
[10:11:34] <tsnag>	 2014/02/04 10:08 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (110)
[10:12:34] <tsnag>	 2014/02/04 10:11 WARN z-dat-s4-a MySQL slave No slaves defined
[10:12:34] <tsnag>	 2014/02/04 10:08 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110)
[10:12:34] <tsnag>	 2014/02/04 10:11 WARN z-dat-s7-a /sql DISK WARNING - free space: /sql 65279 MB (9% inode=99%):
[10:12:34] <tsnag>	 2014/02/04 10:08 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (110)
[10:13:34] <tsnag>	 2014/02/04 10:12  OK  z-dat-s6-a MySQL Uptime: 35796  Threads: 7  Questions: 8036679  Slow queries: 2921  Opens: 3273  Flush tables: 1  Open tables: 3223  Queries per second avg: 224.513
[10:13:34] <tsnag>	 2014/02/04 10:12  OK  z-dat-s7-a MySQL Uptime: 35795  Threads: 18  Questions: 2819022  Slow queries: 2171  Opens: 4115  Flush tables: 1  Open tables: 4008  Queries per second avg: 78.754
[10:13:34] <tsnag>	 2014/02/04 10:12  OK  z-dat-s7-a MySQL slave Uptime: 35795  Threads: 19  Questions: 2819018  Slow queries: 2171  Opens: 4115  Flush tables: 1  Open tables: 4008  Queries per second avg: 78.754 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 272
[10:13:34] <tsnag>	 2014/02/04 10:12  OK  z-dat-s7-a SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)
[10:14:34] <tsnag>	 2014/02/04 10:13 WARN cassia /sql DISK WARNING - free space: /sql 71248 MB (6% inode=99%):
[10:15:34] <tsnag>	 2014/02/04 10:14 CRIT cassia /sql DISK CRITICAL - free space: /sql 70896 MB (5% inode=99%):
[10:35:37] <tsnag>	 2014/02/04 10:34 WARN cassia /sql DISK WARNING - free space: /sql 72017 MB (6% inode=99%):
[10:35:55] <jem->	 willow seems to be unaccesable, anyone around?
[10:36:37] <tsnag>	 2014/02/04 10:35 CRIT cassia /sql DISK CRITICAL - free space: /sql 70773 MB (5% inode=99%):
[11:01:38] <tsnag>	 2014/02/04 11:00 WARN cassia /sql DISK WARNING - free space: /sql 71706 MB (6% inode=99%):
[11:02:39] <tsnag>	 2014/02/04 11:01 CRIT cassia /sql DISK CRITICAL - free space: /sql 70849 MB (5% inode=99%):
[11:05:39] <tsnag>	 2014/02/04 11:04 WARN cassia /sql DISK WARNING - free space: /sql 71023 MB (6% inode=99%):
[11:06:40] <tsnag>	 2014/02/04 11:05 CRIT cassia /sql DISK CRITICAL - free space: /sql 70867 MB (5% inode=99%):
[11:10:39] <tsnag>	 2014/02/04 11:07 CRIT z-dat-s7-a SSH CRITICAL - Socket timeout after 10 seconds
[11:12:40] <tsnag>	 2014/02/04 11:09 CRIT z-dat-s4-a SMTP CRITICAL - Socket timeout after 10 seconds
[11:12:40] <tsnag>	 2014/02/04 11:08 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110)
[11:12:40] <tsnag>	 2014/02/04 11:09 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (110)
[11:13:40] <tsnag>	 2014/02/04 11:09 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (110)
[11:14:40] <tsnag>	 2014/02/04 11:14  OK  z-dat-s4-a SMTP SMTP OK - 0.001 sec. response time
[11:14:40] <tsnag>	 2014/02/04 11:07 CRIT z-dat-s6-a SSH CRITICAL - Socket timeout after 10 seconds
[11:14:40] <tsnag>	 2014/02/04 11:14  OK  z-dat-s7-a SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)
[11:15:40] <tsnag>	 2014/02/04 11:14  OK  z-dat-s6-a MySQL Uptime: 39516  Threads: 5  Questions: 8591897  Slow queries: 3129  Opens: 3275  Flush tables: 1  Open tables: 3225  Queries per second avg: 217.428
[11:15:40] <tsnag>	 2014/02/04 11:14  OK  z-dat-s6-a SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)
[11:15:40] <tsnag>	 2014/02/04 11:14  OK  z-dat-s7-a MySQL Uptime: 39527  Threads: 22  Questions: 3384050  Slow queries: 2385  Opens: 4121  Flush tables: 1  Open tables: 4014  Queries per second avg: 85.613
[11:15:40] <tsnag>	 2014/02/04 11:14  OK  z-dat-s7-a MySQL slave Uptime: 39527  Threads: 18  Questions: 3384058  Slow queries: 2385  Opens: 4121  Flush tables: 1  Open tables: 4014  Queries per second avg: 85.613 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 316
[11:26:42] <tsnag>	 2014/02/04 11:20 CRIT yarrow aliasd Connection refused
[11:33:40] <tsnag>	 2014/02/04 11:32 WARN cassia /sql DISK WARNING - free space: /sql 71654 MB (6% inode=99%):
[11:34:40] <tsnag>	 2014/02/04 11:33 CRIT cassia /sql DISK CRITICAL - free space: /sql 70798 MB (5% inode=99%):
[12:24:49] <tsnag>	 2014/02/04 12:24 WARN cassia /sql DISK WARNING - free space: /sql 71046 MB (6% inode=99%):
[12:25:49] <tsnag>	 2014/02/04 12:25 CRIT cassia /sql DISK CRITICAL - free space: /sql 70715 MB (5% inode=99%):
[13:05:53] <tsnag>	 2014/02/04 13:05 WARN cassia /sql DISK WARNING - free space: /sql 71879 MB (6% inode=99%):
[13:06:53] <tsnag>	 2014/02/04 13:06 CRIT cassia /sql DISK CRITICAL - free space: /sql 70635 MB (5% inode=99%):
[13:47:57] <tsnag>	 2014/02/04 13:41 WARN wolfsbane / DISK WARNING - free space: / 6227 MB (20% inode=93%):
[14:05:57] <tsnag>	 2014/02/04 14:05 WARN cassia /sql DISK WARNING - free space: /sql 71315 MB (6% inode=99%):
[14:06:57] <tsnag>	 2014/02/04 14:06 CRIT cassia /sql DISK CRITICAL - free space: /sql 70493 MB (5% inode=99%):
[14:17:58] <tsnag>	 2014/02/04 14:11 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2076
[14:46:58] <tsnag>	 2014/02/04 14:46 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3614
[14:49:58] <tsnag>	 2014/02/04 14:49 WARN cassia /sql DISK WARNING - free space: /sql 71377 MB (6% inode=99%):
[15:36:04] <tsnag>	 2014/02/04 15:35 CRIT cassia /sql DISK CRITICAL - free space: /sql 70500 MB (5% inode=99%):
[15:37:04] <tsnag>	 2014/02/04 15:36 WARN cassia /sql DISK WARNING - free space: /sql 71761 MB (6% inode=99%):
[16:32:05] <tsnag>	 2014/02/04 16:31 CRIT cassia /sql DISK CRITICAL - free space: /sql 70979 MB (5% inode=99%):
[16:33:05] <tsnag>	 2014/02/04 16:32 WARN cassia /sql DISK WARNING - free space: /sql 71324 MB (6% inode=99%):
[16:38:06] <tsnag>	 2014/02/04 16:37 CRIT cassia /sql DISK CRITICAL - free space: /sql 70924 MB (5% inode=99%):
[16:40:07] <tsnag>	 2014/02/04 16:39 WARN cassia /sql DISK WARNING - free space: /sql 71946 MB (6% inode=99%):
[17:07:06] <tsnag>	 2014/02/04 17:06 CRIT cassia /sql DISK CRITICAL - free space: /sql 70883 MB (5% inode=99%):
[17:09:06] <tsnag>	 2014/02/04 17:08 WARN cassia /sql DISK WARNING - free space: /sql 71508 MB (6% inode=99%):
[17:16:07] <tsnag>	 2014/02/04 17:15 CRIT cassia /sql DISK CRITICAL - free space: /sql 71003 MB (5% inode=99%):
[17:21:14] <tsnag>	 2014/02/04 17:20 WARN cassia /sql DISK WARNING - free space: /sql 72291 MB (6% inode=99%):
[17:25:07] <tsnag>	 2014/02/04 17:24 CRIT cassia /sql DISK CRITICAL - free space: /sql 70548 MB (5% inode=99%):
[17:27:07] <tsnag>	 2014/02/04 17:26 WARN cassia /sql DISK WARNING - free space: /sql 71018 MB (6% inode=99%):
[17:32:08] <tsnag>	 2014/02/04 17:31 CRIT cassia /sql DISK CRITICAL - free space: /sql 71002 MB (5% inode=99%):
[17:37:47] <wikirc>	 [[Special:Log/newusers]] create 10 * Rebeca731 *  (New user account)
[17:48:07] <tsnag>	 2014/02/04 17:47 CRIT wolfsbane / DISK CRITICAL - free space: / 3286 MB (10% inode=93%):
[17:50:07] <tsnag>	 2014/02/04 17:49 WARN cassia /sql DISK WARNING - free space: /sql 71010 MB (6% inode=99%):
[17:53:04] <tsnag>	 2014/02/04 17:51 CRIT cassia /sql DISK CRITICAL - free space: /sql 71002 MB (5% inode=99%):
[17:59:07] <tsnag>	 2014/02/04 17:58 WARN cassia /sql DISK WARNING - free space: /sql 71160 MB (6% inode=99%):
[18:00:09] <tsnag>	 2014/02/04 17:59 CRIT cassia /sql DISK CRITICAL - free space: /sql 70591 MB (5% inode=99%):
[18:00:09] <tsnag>	 2014/02/04 18:00 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3395
[18:04:09] <tsnag>	 2014/02/04 18:03 WARN cassia /sql DISK WARNING - free space: /sql 72225 MB (6% inode=99%):
[18:04:09] <tsnag>	 2014/02/04 18:04  OK  z-dat-s2-b MySQL slave Uptime: 1482039  Threads: 12  Questions: 316780269  Slow queries: 14121  Opens: 4458437  Flush tables: 1  Open tables: 256  Queries per second avg: 213.746 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1530
[18:05:09] <tsnag>	 2014/02/04 18:04 CRIT cassia /sql DISK CRITICAL - free space: /sql 70986 MB (5% inode=99%):
[18:11:10] <tsnag>	 2014/02/04 18:10 WARN cassia /sql DISK WARNING - free space: /sql 71125 MB (6% inode=99%):
[18:12:10] <tsnag>	 2014/02/04 18:11 CRIT cassia /sql DISK CRITICAL - free space: /sql 70971 MB (5% inode=99%):
[19:23:15] <tsnag>	 2014/02/04 19:22 WARN cassia /sql DISK WARNING - free space: /sql 71055 MB (6% inode=99%):
[19:24:15] <tsnag>	 2014/02/04 19:23 CRIT cassia /sql DISK CRITICAL - free space: /sql 70834 MB (5% inode=99%):
[19:24:50] <wikirc>	 [[Special:Log/newusers]] create 10 * Fallon578 *  (New user account)
[19:46:05] <wikirc>	 [[Special:Log/newusers]] create 10 * Superyetkin *  (New user account)
[20:02:22] <tsnag>	 2014/02/04 20:00 WARN cassia /sql DISK WARNING - free space: /sql 71979 MB (6% inode=99%):
[20:02:41] <tsnag>	 2014/02/04 20:01 CRIT cassia /sql DISK CRITICAL - free space: /sql 70748 MB (5% inode=99%):
[20:32:17] <tsnag>	 2014/02/04 20:31 WARN cassia /sql DISK WARNING - free space: /sql 71915 MB (6% inode=99%):
[20:33:21] <tsnag>	 2014/02/04 20:32 CRIT cassia /sql DISK CRITICAL - free space: /sql 70668 MB (5% inode=99%):
[21:01:19] <tsnag>	 2014/02/04 21:00 WARN cassia /sql DISK WARNING - free space: /sql 71821 MB (6% inode=99%):
[21:02:19] <tsnag>	 2014/02/04 21:01 CRIT cassia /sql DISK CRITICAL - free space: /sql 70588 MB (5% inode=99%):
[22:26:20] <tsnag>	 2014/02/04 22:20 CRIT nightshade aliasd Connection refused
[23:10:22] <tsnag>	 2014/02/04 23:09 WARN cassia /sql DISK WARNING - free space: /sql 71984 MB (6% inode=99%):
[23:11:22] <tsnag>	 2014/02/04 23:10 CRIT cassia /sql DISK CRITICAL - free space: /sql 70355 MB (5% inode=99%):
[23:18:33] <tsnag>	 2014/02/04 23:12 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2100
[23:34:23] <tsnag>	 2014/02/04 23:33 WARN cassia /sql DISK WARNING - free space: /sql 71320 MB (6% inode=99%):
[23:46:21] <dschwen>	 oh my, toolserver is a mess for me right now
[23:46:42] <dschwen>	 /mnt/user-storage is not responsive
[23:48:24] <tsnag>	 2014/02/04 23:48 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3608
[23:49:01] <scfc_de>	 dschwen: It's not very fast, but I can access it from yarrow.
[23:49:22] <dschwen>	 hm, ok, I tried nightshade and willow
[23:49:42] <dschwen>	 sucks if you want to migrate stuff off of it
[23:51:22] <scfc_de>	 I wouldn't disagree :-).  Would be nice to have stuff fixed with two paid admins (and noone else).