[00:06:54] / on willow is WARNING: DISK WARNING - free space: / 21700 MB (20% inode=99%): [00:06:54] /tmp on willow is WARNING: DISK WARNING - free space: / 21700 MB (20% inode=99%): [00:08:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 567810 MB (10% inode=46%): [00:17:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:17:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:18:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:44:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:45:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:49:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:58:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:59:53] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 114520 MB (11% inode=99%): [01:06:53] / on willow is WARNING: DISK WARNING - free space: / 21570 MB (20% inode=99%): [01:06:53] /tmp on willow is WARNING: DISK WARNING - free space: / 21570 MB (20% inode=99%): [01:08:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 564401 MB (10% inode=46%): [01:17:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:17:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:18:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:44:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:45:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:49:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:58:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:07:53] / on willow is WARNING: DISK WARNING - free space: / 21435 MB (20% inode=98%): [02:07:54] /tmp on willow is WARNING: DISK WARNING - free space: / 21435 MB (20% inode=98%): [02:08:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 561741 MB (10% inode=46%): [02:17:21] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:18:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:19:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:44:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:45:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:49:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:58:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:07:53] / on willow is WARNING: DISK WARNING - free space: / 21193 MB (20% inode=98%): [03:08:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21288 MB (20% inode=98%): [03:09:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 556366 MB (10% inode=46%): [03:17:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:18:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:19:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:45:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:49:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:58:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:07:53] / on willow is WARNING: DISK WARNING - free space: / 21066 MB (20% inode=98%): [04:08:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21037 MB (20% inode=98%): [04:09:21] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:09:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552987 MB (10% inode=45%): [04:10:44] We need a better CAPTCHA on the wiki [04:14:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:17:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:18:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:19:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:45:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:49:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:58:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:07:53] / on willow is WARNING: DISK WARNING - free space: / 20994 MB (20% inode=98%): [05:08:53] /tmp on willow is WARNING: DISK WARNING - free space: / 20992 MB (20% inode=98%): [05:09:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551484 MB (10% inode=45%): [05:17:21] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:19:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:20:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:41:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:45:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:50:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:54:57] question, if i have a user db called 'u_legoktm_stuff_p', i know the _p lets other users access it, but is there a way I can have another user write to it? [05:55:08] [[tswiki:Database access]] isnt very clear about this [05:59:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:59:52] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 11183 MB (0% inode=94%): [06:08:52] / on willow is WARNING: DISK WARNING - free space: / 20730 MB (19% inode=98%): [06:09:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20718 MB (19% inode=98%): [06:10:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551214 MB (10% inode=45%): [06:17:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:20:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:20:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:34:53] Load avg. on adenia is WARNING: WARNING - load average: 15.12, 14.95, 9.89 [06:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:36:53] Load avg. on adenia is OK: OK - load average: 14.82, 14.92, 10.51 [06:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:39:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:41:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551043 MB (10% inode=45%): [06:41:32] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 70 MB (16% inode=98%): [06:42:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:43:33] /tmp on hemlock is OK: DISK OK - free space: /tmp 102 MB (22% inode=98%): [06:46:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:46:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:50:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:51:31] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 76 MB (17% inode=98%): [06:59:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:08:53] / on willow is WARNING: DISK WARNING - free space: / 20212 MB (19% inode=98%): [07:09:54] /tmp on willow is WARNING: DISK WARNING - free space: / 20209 MB (19% inode=98%): [07:17:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:20:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:20:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:33:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:34:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550704 MB (10% inode=45%): [07:36:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:39:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:39:35] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:42:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:46:13] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:46:14] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:46:14] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:46:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:49:03] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:51:02] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:02] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:03] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:03] / on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:12] RAID on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:13] Environment IPMI on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:32] /tmp on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:33] /sql on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:41] Load avg. on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:53] / on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:53] /tmp on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:53] Load avg. on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:53] Load avg. on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:53] /tmp on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:53] /tmp on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:54] Load avg. on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:54] / on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:54] / on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:55] /tmp on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:55] Load avg. on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:51:56] / on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:52:03] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:03] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:03] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:03] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:03] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:03] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:21] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [07:53:22] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [07:53:53] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [07:53:53] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [07:54:02] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [07:54:02] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [07:57:32] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:59:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550565 MB (10% inode=45%): [08:02:21] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:22] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:22] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:23] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:23] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:23] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:23] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:23] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:24] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:24] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:25] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:25] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:26] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:42] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:42] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:03:03] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:03:03] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:03:13] Load avg. on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:05:22] / on hyacinth is OK: DISK OK - free space: / 7838 MB (26% inode=85%): [08:05:22] NTP on hyacinth is OK: NTP OK: Offset 0.014256 secs [08:05:32] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 195 MB (100% inode=99%): [08:05:32] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:05:32] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 7838 MB (26% inode=85%): [08:05:41] MySQL on z-dat-s3-a is OK: Uptime: 335619 Threads: 24 Questions: 276854940 Slow queries: 17793 Opens: 4466529 Flush tables: 1 Open tables: 16384 Queries per second avg: 824.908 [08:05:42] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2655 [08:05:43] MySQL on z-dat-s7-a is OK: Uptime: 33611 Threads: 11 Questions: 6190203 Slow queries: 1580 Opens: 48421 Flush tables: 1 Open tables: 4453 Queries per second avg: 184.171 [08:05:43] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2659 [08:05:52] / on z-dat-s6-a is OK: DISK OK - free space: / 7838 MB (26% inode=85%): [08:05:53] Load avg. on z-dat-s6-a is OK: OK - load average: 0.28, 0.09, 0.40 [08:05:53] Load avg. on z-dat-s7-a is OK: OK - load average: 0.29, 0.09, 0.40 [08:05:54] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 4292 MB (99% inode=99%): [08:05:54] SMTP on z-dat-s7-a is OK: SMTP OK - 0.045 sec. response time [08:05:54] SMTP on z-dat-s4-a is OK: SMTP OK - 0.021 sec. response time [08:05:54] SMTP on z-dat-s6-a is OK: SMTP OK - 0.103 sec. response time [08:05:54] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:05:54] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 9593 MB (99% inode=99%): [08:05:55] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42245 MB (10% inode=99%): [08:05:55] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 82341 MB (8% inode=98%): [08:05:56] Load avg. on z-dat-s4-a is OK: OK - load average: 0.31, 0.09, 0.40 [08:05:56] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 82341 MB (8% inode=98%): [08:05:57] / on z-dat-s7-a is OK: DISK OK - free space: / 7838 MB (26% inode=85%): [08:06:13] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:06:22] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [08:06:42] Load avg. on hyacinth is OK: OK - load average: 0.73, 0.25, 0.44 [08:06:52] /tmp on hyacinth is OK: DISK OK - free space: /tmp 5638 MB (100% inode=99%): [08:08:52] / on willow is WARNING: DISK WARNING - free space: / 20028 MB (19% inode=98%): [08:10:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20023 MB (19% inode=98%): [08:12:13] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:15:32] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 79 MB (18% inode=98%): [08:17:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:24:12] MySQL on z-dat-s3-a is OK: Uptime: 1085 Threads: 13 Questions: 400 Slow queries: 3 Opens: 193 Flush tables: 1 Open tables: 184 Queries per second avg: 0.368 [08:25:52] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3613 [08:28:22] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:22] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:22] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 82408 MB (8% inode=98%): [08:28:54] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42216 MB (10% inode=99%): [08:28:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 82408 MB (8% inode=98%): [08:29:42] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3601 [08:33:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:33:32] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:34:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550374 MB (10% inode=45%): [08:34:33] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 93 MB (20% inode=98%): [08:36:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:39:29] @replag [08:39:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:39:33] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:39:36] Merlissimo: s3-rr-a: 1h 18m 9s [+1.00 s/s]; s3-user: 1h 18m 14s [+1.00 s/s]; s6-rr-a: 1h 10m 55s [+0.82 s/s]; s6-user: 1h 10m 55s [+0.82 s/s]; s7-rr-a: 1h 7m 28s [+0.71 s/s]; s7-user: 1h 7m 28s [+0.71 s/s] [08:40:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:42:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:46:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:50:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:59:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550216 MB (10% inode=45%): [08:59:33] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 78 MB (18% inode=98%): [09:00:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:00:21] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:00:22] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:00:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:00:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 82417 MB (8% inode=98%): [09:00:53] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42200 MB (10% inode=99%): [09:00:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 82417 MB (8% inode=98%): [09:06:22] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5930 [09:09:52] / on willow is WARNING: DISK WARNING - free space: / 19859 MB (18% inode=98%): [09:11:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19854 MB (18% inode=98%): [09:17:33] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:20:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:21:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:25:52] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6164 [09:29:41] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5963 [09:33:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:33:32] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:35:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 549901 MB (10% inode=45%): [09:35:33] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 77 MB (17% inode=98%): [09:36:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:39:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:39:33] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:40:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:42:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:46:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:50:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:59:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 549725 MB (10% inode=45%): [09:59:32] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 76 MB (17% inode=98%): [10:00:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:06:22] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6990 [10:09:53] / on willow is WARNING: DISK WARNING - free space: / 19756 MB (18% inode=98%): [10:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19750 MB (18% inode=98%): [10:12:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:12:32] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:17:33] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:17:53] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:18:02] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:18:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:18:03] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:18:33] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:18:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:18:53] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3460 [10:19:03] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:21:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:26:54] MySQL slave on z-dat-s6-a is OK: Uptime: 42079 Threads: 11 Questions: 4788554 Slow queries: 3856 Opens: 93535 Flush tables: 1 Open tables: 2562 Queries per second avg: 113.799 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1794 [10:29:32] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 66 MB (15% inode=98%): [10:29:33] CAM on hemlock is OK: OK - cam detected no new errors [10:29:33] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:29:42] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6100 [10:29:53] SMTP on hemlock is OK: SMTP OK - 0.003 sec. response time [10:29:53] / on hemlock is OK: DISK OK - free space: / 6370 MB (31% inode=89%): [10:30:02] /home on hemlock is OK: DISK OK - free space: /home 19996 MB (39% inode=87%): [10:30:02] Load avg. on hemlock is OK: OK - load average: 0.10, 0.14, 0.16 [10:30:02] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [10:30:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 549453 MB (10% inode=45%): [10:33:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:33:32] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:34:42] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3205 [10:36:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 549399 MB (10% inode=45%): [10:36:32] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 82 MB (18% inode=98%): [10:37:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:37:42] MySQL slave on z-dat-s7-a is OK: Uptime: 42727 Threads: 11 Questions: 6708746 Slow queries: 2995 Opens: 48638 Flush tables: 1 Open tables: 4648 Queries per second avg: 157.14 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1390 [10:39:22] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3570 [10:40:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:40:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:42:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:42:43] Getting mySQL errors on this random article toolserver tool - is this a widespread problem, or just this tool? https://toolserver.org/~erwin85/randomarticle.php?lang=en&family=wikipedia&categories=Unreferenced+BLPs&subcats=1&d=2 [10:43:35] ok, I see db-cluster: sql-s4-user ErrorDOWN [10:43:37] sorry [10:44:05] @replag [10:44:05] Danny_B|backup: s2-user: 34m 48s [+0.69 s/s]; s2-user-c: 30s [-0.06 s/s]; s3-rr-a: 44m 9s [-2.69 s/s]; s3-user: 44m 9s [-2.69 s/s]; s5-user-c: 30s [-0.06 s/s] [10:44:58] if replag was a band, would it be dirge, emo, shoegaze, or what? [10:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:47:22] MySQL slave on z-dat-s3-a is OK: Uptime: 9678 Threads: 15 Questions: 5692463 Slow queries: 2096 Opens: 88887 Flush tables: 1 Open tables: 16384 Queries per second avg: 588.185 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1529 [10:47:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:50:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:56:02] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:56:53] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:57:02] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:57:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:57:33] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:57:41] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:01:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:01:33] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:06:03] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:53] / on willow is WARNING: DISK WARNING - free space: / 19619 MB (18% inode=98%): [11:11:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19614 MB (18% inode=98%): [11:16:53] SMTP on hemlock is OK: SMTP OK - 0.002 sec. response time [11:16:53] / on hemlock is OK: DISK OK - free space: / 6369 MB (31% inode=89%): [11:17:02] /home on hemlock is OK: DISK OK - free space: /home 20079 MB (40% inode=87%): [11:17:02] Load avg. on hemlock is OK: OK - load average: 0.27, 0.28, 0.33 [11:17:13] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [11:17:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 543498 MB (10% inode=45%): [11:17:33] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 65 MB (15% inode=98%): [11:17:33] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:17:33] CAM on hemlock is OK: OK - cam detected no new errors [11:18:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:20:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:21:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:30:06] where's source code of irc bots in this channel? [11:33:26] exactly I want code of reba but where is it now? [11:33:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:33:33] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:38:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:39:32] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:39:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:53] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:40:02] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:40:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:40:03] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:40:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:40:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:42:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:43:02] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:48:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:50:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:00:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:09:53] / on willow is WARNING: DISK WARNING - free space: / 19514 MB (18% inode=98%): [12:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19512 MB (18% inode=98%): [12:19:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:27:53] / on hemlock is OK: DISK OK - free space: / 6365 MB (31% inode=89%): [12:28:02] Load avg. on hemlock is OK: OK - load average: 0.32, 0.32, 0.29 [12:28:03] /home on hemlock is OK: DISK OK - free space: /home 19060 MB (38% inode=87%): [12:28:13] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [12:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 528039 MB (9% inode=44%): [12:28:33] /tmp on hemlock is OK: DISK OK - free space: /tmp 731 MB (67% inode=98%): [12:28:33] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:28:33] CAM on hemlock is OK: OK - cam detected no new errors [12:31:14] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:32:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:38:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:40:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:42:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:43:54] SMTP on hemlock is CRITICAL: Connection refused [12:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:48:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:50:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:59:53] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 114089 MB (11% inode=99%): [13:00:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:08:33] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2175.000000 [13:09:54] / on willow is WARNING: DISK WARNING - free space: / 19420 MB (18% inode=98%): [13:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19415 MB (18% inode=98%): [13:19:33] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:22:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 523294 MB (9% inode=44%): [13:33:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:38:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:38:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [13:41:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:43:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:44:53] SMTP on hemlock is CRITICAL: Connection refused [13:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:48:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:50:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:00:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:01:03] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2140 [14:01:14] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2145.000000 [14:10:52] / on willow is WARNING: DISK WARNING - free space: / 19307 MB (18% inode=98%): [14:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19305 MB (18% inode=98%): [14:19:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:22:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:28:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [14:28:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3621.000000 [14:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 520348 MB (9% inode=44%): [14:32:12] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1911.000000 [14:34:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:38:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:38:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6179.000000 [14:41:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:43:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:45:52] SMTP on hemlock is CRITICAL: Connection refused [14:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:49:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:51:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:00:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:10:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:10:52] / on willow is WARNING: DISK WARNING - free space: / 19180 MB (18% inode=98%): [15:11:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19178 MB (18% inode=98%): [15:12:12] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:20:33] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:22:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:28:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6170.000000 [15:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571571 MB (10% inode=46%): [15:29:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6205 [15:30:41] liangent: Hello. [15:31:08] reba is kind of a stupid bot. [15:31:18] Brooke: hello [15:31:28] She just checks a Gmail account every few minutes. [15:31:31] She's also apparently dead... [15:31:58] I am trying to write a kinda inverse weighted random function in python, anyone have any suggestions? [15:32:12] Betacommand: Find a module that does the math for you? [15:32:13] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3008.000000 [15:32:30] You don't want to write that kind of math yourself. [15:32:41] Brooke: Ive been looking [15:32:46] havent seen anything [15:34:37] There she is. [15:36:26] liangent: http://dpaste.de/PfioX/raw/ [15:36:33] There are surely smarter ways to do this, though. [15:36:52] * Brooke snuggles reba. [15:36:53] * reba snuggles Brooke. [15:38:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:38:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7678.000000 [15:39:11] Brooke: hmm but the wikibugs way is not possible without mailserver admin [15:39:26] Probably. [15:39:36] Reading others' mail is usually restricted. [15:40:00] Though if you have control over the receiving account, you can pipe/echo to ircecho.py, maybe. [15:40:13] Or something. [15:40:18] There's some simpler way to do this, in theory. [15:40:24] Though polling works well enough. [15:40:37] It's actually the message reading that needs clean up. [15:40:53] Manually parsing e-mail is retarded. [15:41:12] Brooke: is it possible to have my @toolserver.org mails stored somewhere? [15:41:25] If you remove your .forward file, I think they'll stay local. [15:41:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:42:52] Your home dir is really locked down. [15:43:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:44:33] Brooke: from other users but not from services [15:44:57] Right... [15:45:17] It makes it more difficult to steal code if read perms are disabled. [15:45:53] SMTP on hemlock is CRITICAL: Connection refused [15:46:15] Getting mySQL errors on this random article toolserver tool - is this a widespread problem, or just this tool? https://toolserver.org/~erwin85/randomarticle.php?lang=en&family=wikipedia&categories=Unreferenced+BLPs&subcats=1&d=2 [15:46:17] I see db-cluster: sql-s4-user ErrorDOWN [15:46:19] Is there any clue as to what's wrong? [15:46:30] Brooke: and passwords [15:47:13] maybe I can o+r but not really needed because it works just fine now [15:47:31] I imagine you have like two or three password files. [15:47:40] Unless you put passwords in every file or something. ;-) [15:47:59] Per-file permissions, ftw. [15:48:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [15:48:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:48:27] lexein: sql-s4-user seems fine to me. [15:48:37] At least I can connect to it. [15:48:51] Brooke: and various backup files... [15:49:04] You backup locally? [15:49:14] Brooke: by editors etc [15:49:22] * Brooke shrugs. [15:49:32] lexein: Where are you seeing sql-s4-user? [15:49:46] I see a bunch of warnings at the top of the page. [15:49:48] http://status.toolserver.org/ says db-cluster: sql-s4-user ErrorDOWN - [15:49:50] I don't want to bug anyone, so is this the right place to ask? [15:49:52] Oh. [15:50:02] Just find Erwin on his talk page at nl.wikipedia. [15:50:04] He'll fix it up. [15:50:06] Erwin is great. [15:50:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:50:24] https://nl.wikipedia.org/wiki/Overleg_gebruiker:Erwin [15:50:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:50:36] status.ts.o is probably out of date. [15:50:50] sql-s4-user doesn't seem down to me. [15:50:58] There was something about the -user DBs on the mailing list. [15:51:01] Didn't read it, though. [15:51:52] Brooke (or are you Bree?) - thanks [15:51:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:53:00] * lexein makes an obscure reference to an NZ soap opera - hee [15:53:32] Australian, isn't it? [15:54:03] en.wikipedia claims NZ [15:54:05] hee [15:57:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:58:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:59:00] me recaps: Bree, a doctor, is the long lost put-up-for-adoption sister of Brooke, the troubled overzealous, grasping, even, doctor. Long story short, Bree attempte to kill Brooke and complete her assumption of her identity, even seeming to crazily believe that she _is_ Brooke, only answering to that name. Sooooo, Brooke, watch out for long-lost siblings out to nick your life. [16:00:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:03:13] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3589.000000 [16:05:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:10:53] / on willow is WARNING: DISK WARNING - free space: / 19056 MB (18% inode=98%): [16:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19055 MB (18% inode=98%): [16:20:33] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:22:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:28:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7656.000000 [16:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571942 MB (10% inode=46%): [16:29:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7699 [16:38:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:38:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10590.000000 [16:41:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:42:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:43:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:45:53] SMTP on hemlock is CRITICAL: Connection refused [16:47:12] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:48:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:49:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:50:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:51:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:00:14] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:03:13] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3391.000000 [17:10:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:10:53] / on willow is WARNING: DISK WARNING - free space: / 18931 MB (18% inode=98%): [17:11:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 18928 MB (18% inode=98%): [17:19:01] @replag [17:19:04] Merlissimo: s1-rr-a: 2h 35m 7s [+0.53 s/s]; s1-user: 2h 35m 10s [+0.53 s/s]; s2-user: 5h 41m 30s [+0.85 s/s]; s2-user-c: 3h 21m 16s [+0.68 s/s]; s3-rr-a: 59s [+0.00 s/s]; s3-user: 59s [+0.00 s/s]; s5-user-c: 3h 21m 16s [+0.68 s/s] [17:20:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:20:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:22:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:28:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571581 MB (10% inode=46%): [17:29:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9299 [17:29:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9303.000000 [17:36:13] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1745.000000 [17:38:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:38:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12687.000000 [17:42:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:43:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:45:54] SMTP on hemlock is CRITICAL: Connection refused [17:49:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:50:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:52:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:59:53] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 10601 MB (0% inode=94%): [18:00:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:10:53] / on willow is WARNING: DISK WARNING - free space: / 18810 MB (17% inode=98%): [18:11:54] /tmp on willow is WARNING: DISK WARNING - free space: / 18808 MB (17% inode=98%): [18:12:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:21:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:22:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:25:21] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:26:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:28:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571221 MB (10% inode=46%): [18:29:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8584 [18:29:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8591.000000 [18:38:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:38:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14181.000000 [18:42:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:43:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:46:54] SMTP on hemlock is CRITICAL: Connection refused [18:49:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:50:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:51:13] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:52:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:52:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:00:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:01:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:07:53] I'm getting a failure DB connection error: User 'platonides' has exceeded the 'max_user_connections' resource (current value: 15) (sql-s2-rr.toolserver.org) [19:08:06] yet I have no idea where those 15 connections may be [19:08:42] I killed my php.fcgi on wolfsbane (none on ortelius) in case they were holding them [19:08:44] but no joy [19:08:46] any idea? [19:10:53] / on willow is WARNING: DISK WARNING - free space: / 18690 MB (17% inode=98%): [19:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 18688 MB (17% inode=98%): [19:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:21:33] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:22:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570911 MB (10% inode=46%): [19:29:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9956.000000 [19:30:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9940 [19:38:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:38:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15931.000000 [19:42:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:42:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:44:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:46:54] SMTP on hemlock is CRITICAL: Connection refused [19:49:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:50:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:52:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:53:15] I'm getting a lot of Segmentation faults-core dumped from my cronie works during the day... is it just me? [20:00:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:10:53] / on willow is WARNING: DISK WARNING - free space: / 18572 MB (17% inode=98%): [20:11:53] SMTP on hemlock is OK: SMTP OK - 0.119 sec. response time [20:12:54] /tmp on willow is OK: DISK OK - free space: / 25533 MB (24% inode=99%): [20:12:54] / on willow is OK: DISK OK - free space: / 25533 MB (24% inode=99%): [20:20:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:21:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:22:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:28:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570722 MB (10% inode=46%): [20:29:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11298.000000 [20:30:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11346 [20:32:13] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2080.000000 [20:32:22] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 11.817 second response time [20:38:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:38:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19045.000000 [20:42:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:42:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:43:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106834 MB (10% inode=99%): [20:44:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:49:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:50:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:52:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:00:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:00:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 81070 MB (8% inode=98%): [21:00:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 81070 MB (8% inode=98%): [21:00:54] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 41237 MB (10% inode=99%): [21:10:13] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:12:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:14:53] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108543 MB (11% inode=99%): [21:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:22:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:22:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106902 MB (10% inode=99%): [21:25:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3617.000000 [21:29:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14494.000000 [21:29:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570509 MB (10% inode=46%): [21:30:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14546 [21:39:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:39:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22094.000000 [21:41:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:42:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:42:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:45:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:50:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:51:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:53:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:00:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:00:21] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:01:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:23:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:25:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6315.000000 [22:26:33] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 71 MB (16% inode=98%): [22:29:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15442.000000 [22:29:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570157 MB (10% inode=46%): [22:31:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15486 [22:33:33] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:33:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:34:33] /tmp on hemlock is OK: DISK OK - free space: /tmp 96 MB (21% inode=98%): [22:34:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570154 MB (10% inode=46%): [22:35:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:36:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:39:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:39:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24387.000000 [22:39:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:43:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:45:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:50:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:51:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:53:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:59:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570098 MB (10% inode=46%): [23:00:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:04:53] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107938 MB (11% inode=99%): [23:05:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106557 MB (10% inode=99%): [23:06:33] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 74 MB (17% inode=98%): [23:13:53] Load avg. on adenia is WARNING: WARNING - load average: 15.82, 15.49, 12.86 [23:20:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:22:33] /tmp on hemlock is OK: DISK OK - free space: /tmp 94 MB (21% inode=98%): [23:23:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:26:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8066.000000 [23:30:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18075.000000 [23:31:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18116 [23:31:33] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 92 MB (20% inode=98%): [23:33:33] /tmp on hemlock is OK: DISK OK - free space: /tmp 105 MB (22% inode=98%): [23:34:54] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 111184 MB (11% inode=99%): [23:35:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:39:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:40:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:40:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 567097 MB (10% inode=46%): [23:40:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24045.000000 [23:44:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:45:53] Load avg. on adenia is OK: OK - load average: 12.25, 14.24, 15.00 [23:46:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:49:36] Interesting, after checking back a year later, qstat now works propertly from all web servers [23:49:37] https://toolserver.org/~krinkle/sandbox_qstat1.php [23:49:38] :) [23:49:46] * Krinkle was just cleaning up some old tmp php files in public_html [23:50:44] that means it is now a whole lot easier to schedule things from php scripts (previously it required either ssh-ing into a login server, which is hard without php_ssh2 installed, or one would have to keep a queue somewhere and crontab/loop from a login server) [23:51:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:52:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:53:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output