[00:09:48] 2014/02/01 00:02 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [00:10:48] 2014/02/01 00:09 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [00:32:06] Dr. Trigon * Re: [Toolserver-l] What will happen with the Toolserver domain? [00:37:50] 2014/02/01 00:37 WARN cassia /sql DISK WARNING - free space: /sql 71168 MB (6% inode=99%): [00:38:50] 2014/02/01 00:38 CRIT cassia /sql DISK CRITICAL - free space: /sql 69955 MB (5% inode=99%): [00:57:52] 2014/02/01 00:57 WARN cassia /sql DISK WARNING - free space: /sql 71172 MB (6% inode=99%): [00:58:52] 2014/02/01 00:58 CRIT cassia /sql DISK CRITICAL - free space: /sql 68304 MB (5% inode=99%): [00:58:52] 2014/02/01 00:52 WARN hemlock / DISK WARNING - free space: / 4193 MB (20% inode=87%): [00:58:52] 2014/02/01 00:52 WARN hemlock /tmp DISK WARNING - free space: / 4193 MB (20% inode=87%): [01:04:17] Whatever it is, nightshade seems to be hanging. ssh to it doesn't work, and in an active session I can't run any commands. [01:14:53] 2014/02/01 01:14 WARN cassia /sql DISK WARNING - free space: /sql 71467 MB (6% inode=99%): [01:15:54] 2014/02/01 01:15 CRIT cassia /sql DISK CRITICAL - free space: /sql 69897 MB (5% inode=99%): [01:24:54] 2014/02/01 01:24 WARN cassia /sql DISK WARNING - free space: /sql 72574 MB (6% inode=99%): [01:25:54] 2014/02/01 01:25 CRIT cassia /sql DISK CRITICAL - free space: /sql 69641 MB (5% inode=99%): [01:33:42] nightshade still inaccessible, yarrow's working fine. [03:10:59] 2014/02/01 03:10 OK hemlock / DISK OK - free space: / 4225 MB (21% inode=87%): [03:10:59] 2014/02/01 03:10 OK hemlock /tmp DISK OK - free space: / 4225 MB (21% inode=87%): [03:46:02] 2014/02/01 03:45 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3598.000000 [03:47:02] 2014/02/01 03:46 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3585 [04:14:04] 2014/02/01 04:13 OK rosemary MySQL slave Uptime: 93662 Threads: 22 Questions: 49904577 Slow queries: 23974 Opens: 5264 Flush tables: 1 Open tables: 2910 Queries per second avg: 532.815 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1786 [04:14:04] 2014/02/01 04:13 OK rosemary s1 replag QUERY OK: 'SELECT ts_rc_age()' returned 1766.000000 [04:55:07] 2014/02/01 04:54 WARN cassia /sql DISK WARNING - free space: /sql 72084 MB (6% inode=99%): [04:56:07] 2014/02/01 04:55 CRIT cassia /sql DISK CRITICAL - free space: /sql 69297 MB (5% inode=99%): [05:47:11] 2014/02/01 05:47 CRIT z-dat-s5-b APT APT CRITICAL: 2 packages available for upgrade (1 critical updates). [06:50:22] nightshade still down, yarrow still fine. [07:53:17] 2014/02/01 07:46 WARN hemlock / DISK WARNING - free space: / 4193 MB (20% inode=87%): [07:53:17] 2014/02/01 07:46 WARN hemlock /tmp DISK WARNING - free space: / 4193 MB (20% inode=87%): [10:21:26] 2014/02/01 10:20 WARN nightshade Load avg. WARNING - load average: 4.41, 3.69, 19.67 [10:26:27] 2014/02/01 10:25 OK nightshade Load avg. OK - load average: 1.56, 2.65, 14.82 [10:37:41] [[Special:Log/newusers]] create 10 * Prince399 * (New user account) [11:23:29] 2014/02/01 11:16 WARN nightshade Load avg. WARNING - load average: 13.93, 17.58, 12.77 [11:25:29] 2014/02/01 11:24 OK nightshade Load avg. OK - load average: 8.81, 14.77, 12.34 [11:37:29] 2014/02/01 11:36 WARN cassia /sql DISK WARNING - free space: /sql 71097 MB (6% inode=99%): [11:38:29] 2014/02/01 11:37 CRIT cassia /sql DISK CRITICAL - free space: /sql 69840 MB (5% inode=99%): [11:42:20] jem-: around? [11:42:43] jem-: your processes seem to kill the performance of willow [11:48:29] 2014/02/01 11:41 WARN nightshade Load avg. WARNING - load average: 26.00, 19.93, 13.90 [11:52:29] 2014/02/01 11:51 CRIT nightshade Load avg. CRITICAL - load average: 31.59, 25.77, 17.61 [12:01:29] 2014/02/01 12:00 WARN cassia /sql DISK WARNING - free space: /sql 71048 MB (6% inode=99%): [12:02:30] 2014/02/01 12:01 CRIT cassia /sql DISK CRITICAL - free space: /sql 69804 MB (5% inode=99%): [12:15:31] 2014/02/01 12:14 WARN nightshade Load avg. WARNING - load average: 6.11, 11.63, 19.77 [12:31:32] 2014/02/01 12:30 CRIT nightshade Load avg. CRITICAL - load average: 30.32, 24.31, 20.70 [13:16:36] 2014/02/01 13:15 WARN cassia /sql DISK WARNING - free space: /sql 71678 MB (6% inode=99%): [13:17:36] 2014/02/01 13:16 CRIT cassia /sql DISK CRITICAL - free space: /sql 69626 MB (5% inode=99%): [13:58:38] 2014/02/01 13:57 WARN cassia /sql DISK WARNING - free space: /sql 73794 MB (6% inode=99%): [13:59:38] 2014/02/01 13:58 CRIT cassia /sql DISK CRITICAL - free space: /sql 69559 MB (5% inode=99%): [14:04:39] 2014/02/01 13:57 WARN wolfsbane / DISK WARNING - free space: / 6223 MB (20% inode=93%): [14:04:39] 2014/02/01 13:57 WARN wolfsbane /tmp DISK WARNING - free space: / 6223 MB (20% inode=93%): [14:22:39] 2014/02/01 14:21 WARN cassia /sql DISK WARNING - free space: /sql 71668 MB (6% inode=99%): [14:23:39] 2014/02/01 14:22 CRIT cassia /sql DISK CRITICAL - free space: /sql 69511 MB (5% inode=99%): [15:36:43] 2014/02/01 15:34 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1913 [15:45:42] 2014/02/01 15:38 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2159 [15:48:42] 2014/02/01 15:41 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2185 [15:53:42] 2014/02/01 15:46 CRIT yarrow aliasd Connection refused [16:02:29] Danny_B|webgate: willow works for me [16:02:46] nightshade seems overloaded, though [16:03:01] top - 16:02:57 up 18:57, 3 users, load average: 342.10, 336.24, 320.94 [16:03:04] lolwut [16:03:28] I think NFS might be borking [16:10:44] 2014/02/01 16:09 CRIT z-dat-s6-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628 [16:10:44] 2014/02/01 16:10 CRIT z-dat-s7-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3627 [16:11:44] 2014/02/01 16:10 CRIT z-dat-s7-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [16:12:44] 2014/02/01 16:11 WARN z-dat-s7-a /sql DISK WARNING - free space: /sql 65665 MB (9% inode=99%): [16:13:45] 2014/02/01 16:12 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3640 [16:26:46] 2014/02/01 16:25 ?? z-dat-s7-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:27:46] 2014/02/01 16:25 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [16:28:31] valhallasw: not for me though :-/ otherwise i'd have my irssi here instead of using webgate [16:28:46] 2014/02/01 16:25 CRIT z-dat-s7-a SSH CRITICAL - Socket timeout after 10 seconds [16:29:25] just tried and again it ended up on the list of active screens [16:29:47] 2014/02/01 16:25 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [16:30:06] iianm, on both servers the next thing displayed is usually quota [16:30:29] which i don't get to now [16:30:37] so it's my primal suspect [16:30:46] 2014/02/01 16:26 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (110) [16:31:46] 2014/02/01 16:27 CRIT z-dat-s4-a SMTP CRITICAL - Socket timeout after 10 seconds [16:32:46] 2014/02/01 16:25 ?? hyacinth / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:32:47] 2014/02/01 16:25 ?? hyacinth /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:32:47] 2014/02/01 16:25 ?? hyacinth Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:32:47] 2014/02/01 16:25 ?? hyacinth Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:32:47] 2014/02/01 16:25 ?? hyacinth RAID CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:33:54] 2014/02/01 16:25 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [16:34:10] Danny_B|webgate: try ctrl-c. I got a shell on nighshade after that [16:34:54] 2014/02/01 16:33 CRIT z-dat-s6-a Load avg. CHECK_NRPE: Socket timeout after 30 seconds. [16:35:08] so do i. but it is not _my_ shell though [16:35:24] (aka different prompt, no aliases, etc...) [16:35:54] 2014/02/01 16:34 OK hyacinth / DISK OK - free space: / 8608 MB (28% inode=86%): [16:35:54] 2014/02/01 16:34 OK hyacinth /tmp DISK OK - free space: /tmp 12488 MB (100% inode=99%): [16:35:54] 2014/02/01 16:35 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [16:35:54] 2014/02/01 16:34 OK hyacinth Load avg. OK - load average: 0.01, 0.16, 0.55 [16:35:54] 2014/02/01 16:35 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:41:03] Sure, because you cut off .bashrc [16:42:02] 2014/02/01 16:41 OK z-dat-s6-a MySQL Uptime: 390 Threads: 4 Questions: 2571 Slow queries: 0 Opens: 117 Flush tables: 1 Open tables: 106 Queries per second avg: 6.592 [16:44:12] 2014/02/01 16:43 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3484 [16:54:03] 2014/02/01 16:53 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3575 [16:59:02] 2014/02/01 16:58 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3564 [17:00:02] 2014/02/01 16:59 OK z-dat-s7-a MySQL slave Uptime: 6322933 Threads: 7 Questions: 1093011808 Slow queries: 406439 Opens: 4045475 Flush tables: 1 Open tables: 5965 Queries per second avg: 172.864 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1689 [17:03:02] 2014/02/01 17:02 OK z-dat-s6-a MySQL slave Uptime: 1609 Threads: 3 Questions: 351712 Slow queries: 123 Opens: 288 Flush tables: 1 Open tables: 277 Queries per second avg: 218.590 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1588 [17:07:02] 2014/02/01 17:06 OK z-dat-s3-a MySQL slave Uptime: 365805 Threads: 19 Questions: 120458289 Slow queries: 18000 Opens: 903872 Flush tables: 1 Open tables: 16385 Queries per second avg: 329.296 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1742 [17:13:02] 2014/02/01 17:04 CRIT daphne s4 replag (Service Check Timed Out) [17:25:02] 2014/02/01 17:24 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 580.000000 [17:26:02] 2014/02/01 17:18 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [17:31:03] 2014/02/01 17:29 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [17:57:05] 2014/02/01 17:56 CRIT wolfsbane / DISK CRITICAL - free space: / 3281 MB (10% inode=93%): [17:57:05] 2014/02/01 17:56 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3281 MB (10% inode=93%): [18:25:06] 2014/02/01 18:16 CRIT daphne s4 replag (Service Check Timed Out) [18:26:06] 2014/02/01 18:18 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [18:27:06] 2014/02/01 18:25 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 379.000000 [18:27:06] 2014/02/01 18:25 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [18:27:23] [[Template talk:DORIP]] ! 10https://wiki.toolserver.org/w/index.php?diff=8328&oldid=4659&rcid=22934 * 166.147.108.33 * (-16) (Undo revision 4659 by [[Special:Contributions/TeleComNasSprVen|TeleComNasSprVen]] ([[User tarVen|talk]])) [19:05:18] [[Special:Log/newusers]] create 10 * Hollis297 * (New user account) [19:06:01] [[Template talk:DORIP]] M 10https://wiki.toolserver.org/w/index.php?diff=8329&oldid=8328&rcid=22936 * Betacommand * (+16) (Reverted edits by [[Special:Contributions/166.147.108.33|166.147.108.33]] ([[User talk:166.147.108.33|talk]]) to last revision by [[User:TeleComNasSprVen|TeleComNasSprVen]]) [19:06:09] 2014/02/01 18:58 CRIT daphne s4 replag (Service Check Timed Out) [19:06:24] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:166.147.108.3310]] with an expiry time of 6 months (anonymous users only, account creation disabled): Inserting nonsense into pages) [19:08:09] 2014/02/01 19:06 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 211.000000 [19:08:10] 2014/02/01 19:07 OK yarrow aliasd TCP OK - 0.001 second response time on port 984 [500 Not found.] [20:20:18] Login to nightshade seems to stall in /etc/profile.d/quota.sh; NFS seems to be working fine, though. [20:22:21] "ls /mnt" works, "ls -l /mnt" doesn't. So the culprit seems to be /mnt/user-store? [20:23:04] A load of 124.18 with the CPU idling 85.9% still looks pretty cool :-). [20:23:18] amette: Can you fix this? ^ [20:37:54] right, i will never get to quota note