[00:00:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:01:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:07:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:13:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:17:52] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 81859 MB (8% inode=98%): [00:22:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:23:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:24:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:28:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:29:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:40:53] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:40:53] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:40:53] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:41:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 81731 MB (8% inode=98%): [00:41:53] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 41387 MB (10% inode=99%): [00:41:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 81731 MB (8% inode=98%): [00:43:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 562518 MB (10% inode=46%): [00:46:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:49:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:49:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:52:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:52:54] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:52:54] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:52:54] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:57:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:59:53] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [00:59:53] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [01:00:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:01:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:07:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:12:54] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 41320 MB (10% inode=99%): [01:12:55] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 81798 MB (8% inode=98%): [01:12:55] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 81798 MB (8% inode=98%): [01:15:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:19:54] MySQL slave on z-dat-s3-a is OK: Uptime: 1554 Threads: 8 Questions: 416 Slow queries: 2 Opens: 175 Flush tables: 1 Open tables: 165 Queries per second avg: 0.267 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1559 [01:19:55] MySQL on z-dat-s3-a is OK: Uptime: 1554 Threads: 7 Questions: 418 Slow queries: 2 Opens: 175 Flush tables: 1 Open tables: 165 Queries per second avg: 0.268 [01:22:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:29:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:31:52] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2053 [01:32:23] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:33:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:34:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:36:53] MySQL slave on z-dat-s3-a is OK: Uptime: 2574 Threads: 25 Questions: 331497 Slow queries: 37 Opens: 12559 Flush tables: 1 Open tables: 12548 Queries per second avg: 128.786 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1656 [01:44:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 562433 MB (10% inode=46%): [01:46:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:49:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:49:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:52:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:58:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:00:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:01:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:07:22] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:10:53] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109344 MB (11% inode=99%): [02:22:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:25:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:29:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:35:21] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:36:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:40:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:44:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 560409 MB (10% inode=46%): [02:46:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:49:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:49:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:52:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:58:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:00:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:02:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:07:22] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:20:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:23:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:29:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:44:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559473 MB (10% inode=46%): [03:47:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:49:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:49:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:52:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:58:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:00:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:03:41] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1883.000000 [04:03:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:04:10] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1889 [04:04:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:40] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [04:07:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:08:11] MySQL slave on rosemary is OK: Uptime: 14215847 Threads: 121 Questions: 6204384280 Slow queries: 1860156 Opens: 235741 Flush tables: 6 Open tables: 3819 Queries per second avg: 436.441 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [04:19:20] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:20:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:23:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:29:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:44:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 560226 MB (10% inode=46%): [04:47:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:50:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:50:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:53:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:58:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:01:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:03:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:07:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:17:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106870 MB (10% inode=99%): [05:20:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:23:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:51] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107453 MB (11% inode=99%): [05:30:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:45:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 560040 MB (10% inode=46%): [05:48:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:51:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:51:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:54:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:58:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:01:01] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 4314 MB (0% inode=92%): [06:01:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:04:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:07:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:20:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:30:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:40] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:40] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:45:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559754 MB (10% inode=46%): [06:48:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:51:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:51:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:55:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:58:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:02:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:04:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:08:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:12:01] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 12.442 second response time [07:21:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:30:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:31:42] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:42] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:31:42] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:40:51] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:41:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:45:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559675 MB (10% inode=46%): [07:51:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:51:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:55:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:57:01] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:57:01] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:57:01] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:58:30] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:58:31] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:58:31] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:58:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:59:10] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 82839 MB (8% inode=98%): [07:59:10] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 41995 MB (10% inode=99%): [07:59:10] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 82839 MB (8% inode=98%): [08:02:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:03:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:03:52] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:03:52] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:04:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:05:11] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.701 second response time [08:08:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:09:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:21:25] Hercule Hercule * [Toolserver-l] Trouble with shell file encoding [08:22:52] MySQL on z-dat-s3-a is OK: Uptime: 1416 Threads: 1 Questions: 25 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 15 Queries per second avg: 0.17 [08:22:52] MySQL slave on z-dat-s3-a is OK: Uptime: 1416 Threads: 1 Questions: 26 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 15 Queries per second avg: 0.18 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1697 [08:30:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:31:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:52] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:31:52] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2074 [08:32:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:34:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2042.000000 [08:40:51] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:40:51] MySQL slave on z-dat-s3-a is OK: Uptime: 2496 Threads: 14 Questions: 209809 Slow queries: 29 Opens: 10853 Flush tables: 1 Open tables: 10129 Queries per second avg: 84.58 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1645 [08:41:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:46:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559307 MB (10% inode=46%): [08:47:25] Wolfgang ten Weges * Re: [Toolserver-l] Trouble with shell file encoding [08:51:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:51:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:55:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:59:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:02:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:04:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:09:50] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:17:25] Danny B. * Re: [Toolserver-l] Trouble with shell file encoding [09:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3603.000000 [09:31:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:32:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:32:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:32:52] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:42:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:46:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559094 MB (10% inode=46%): [09:50:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3580.000000 [09:51:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:52:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:56:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:59:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:00:51] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:01:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:02:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:05:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:09:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:15:25] Hercule Hercule * Re: [Toolserver-l] Trouble with shell file encoding [10:22:51] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1713.000000 [10:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:29:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105477 MB (10% inode=99%): [10:31:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:32:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:32:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:32:51] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:33:52] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109300 MB (11% inode=99%): [10:46:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558756 MB (10% inode=46%): [10:52:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:52:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:53:25] DaB. * Re: [Toolserver-l] Trouble with shell file encoding [10:53:32] hello all [10:56:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:59:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:01:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:01:51] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2198 [11:02:11] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2170 [11:02:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:05:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:09:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:13:23] http://status.toolserver.org/ shows [11:13:50] http://toolserver.org/~daniel/WikiSense/CategoryIntersect.php shows s6 failed [11:14:28] what does "db-cluster: sql-s6-rr OK" really mean? [11:15:40] I'll drop soon, but if u no the answer, post it here pls. I'll check it in logs [11:16:08] that the database-cluster s6 (round-robing-variant) ist working [11:16:19] and it is working for me [11:17:11] hm... weird [11:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:26:52] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3658 [11:27:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3656 [11:32:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:32:51] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:32:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:32:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:43:50] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2173 [11:47:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558680 MB (10% inode=46%): [11:50:26] Hercule Hercule * Re: [Toolserver-l] Trouble with shell file encoding [11:52:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:52:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:57:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:59:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:01:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:02:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:05:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:07:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3606 [12:09:25] emijrp * Re: [Toolserver-l] Web servers unresponsive [12:10:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:11:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2067.000000 [12:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:51] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7218 [12:28:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7144 [12:30:25] DaB. * Re: [Toolserver-l] Web servers unresponsive [12:31:50] /aux0 on hemlock is CRITICAL: NRPE: Unable to read output [12:32:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558600 MB (10% inode=46%): [12:33:25] DaB. * [Toolserver-l] Restarted query-killer [12:33:39] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:33:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:33:51] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:33:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:35:02] I think this toolserver requirs goddamn big lot of efforts to maintain [12:35:52] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:41:51] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:53:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:53:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:57:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:59:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:01:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:03:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:05:26] emijrp * Re: [Toolserver-l] Web servers unresponsive [13:05:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:07:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7022 [13:09:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558415 MB (10% inode=46%): [13:10:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:11:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2167.000000 [13:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:27:51] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10780 [13:28:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10663 [13:33:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:33:51] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:33:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:54:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:54:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:57:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:59:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:02:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:04:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:05:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:07:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10556 [14:08:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 103219 MB (10% inode=99%): [14:09:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558379 MB (10% inode=46%): [14:09:52] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 111224 MB (11% inode=99%): [14:10:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:10:51] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1798.000000 [14:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:27:51] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14360 [14:28:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14192 [14:34:41] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:34:52] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:34:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:35:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2125.000000 [14:51:26] Platonides * Re: [Toolserver-l] Web servers unresponsive [14:55:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:55:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:57:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:59:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:02:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:05:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:05:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:07:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14123 [15:10:50] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:10:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558258 MB (10% inode=46%): [15:11:26] DaB. * Re: [Toolserver-l] Web servers unresponsive [15:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:27:52] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17909 [15:29:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17750 [15:35:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:35:51] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:35:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3054.000000 [15:35:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:55:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:55:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:58:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:00:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:03:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:05:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:05:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:08:26] Krinkle * [Toolserver-l] qcronsub warning: "Please add maximum runtime" [16:08:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17658 [16:10:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:10:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558136 MB (10% inode=46%): [16:23:26] Platonides * Re: [Toolserver-l] qcronsub warning: "Please add maximum runtime" [16:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:27:52] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21429 [16:29:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21051 [16:29:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:30:12] toolserver is still bad? :) [16:32:27] Krinkle * Re: [Toolserver-l] qcronsub warning: "Please add maximum runtime" [16:35:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:35:51] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:35:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2094.000000 [16:35:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:54:00] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:55:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:55:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:58:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:00:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:03:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:05:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:05:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:07:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [17:09:50] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21194 [17:11:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:11:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 557991 MB (10% inode=46%): [17:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:28:50] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25010 [17:29:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24155 [17:35:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:35:52] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:35:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:41:29] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:41:30] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:41:30] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:42:01] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 82199 MB (8% inode=98%): [17:42:01] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 40213 MB (9% inode=99%): [17:42:01] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 82199 MB (8% inode=98%): [17:54:30] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:54:30] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:56:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:56:42] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:58:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:58:50] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3560.000000 [18:00:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:01:00] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 15173 MB (1% inode=95%): [18:03:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:05:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:05:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:09:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24602 [18:12:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 557897 MB (10% inode=46%): [18:12:50] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:14:01] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 40185 MB (9% inode=99%): [18:14:01] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 82217 MB (8% inode=98%): [18:25:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:29:00] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28546 [18:29:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26860 [18:35:40] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:36:52] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:36:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:57:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:57:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:58:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:58:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2918.000000 [19:00:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:04:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:06:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:06:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:10:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27833 [19:12:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 557779 MB (10% inode=46%): [19:12:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:25:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:29:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26412 [19:29:51] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32133 [19:36:40] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:37:26] Platonides * [Toolserver-l] Very slow connections to the sql servers [19:37:50] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:37:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:57:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:57:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:59:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:59:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2244.000000 [20:00:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:04:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:07:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:07:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:10:51] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30891 [20:12:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 557671 MB (10% inode=46%): [20:12:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:25:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:29:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28484 [20:29:51] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35621 [20:36:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:37:52] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:37:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default