[00:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:02:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:03:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:04:18] / on wolfsbane is WARNING: DISK WARNING - free space: / 6256 MB (20% inode=93%): [00:04:40] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89437 MB (14% inode=99%): [00:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:22:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:43:32] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6253 MB (20% inode=93%): [00:58:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8612.000000 [00:58:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8617 [01:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:02:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:02:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:03:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:04:19] / on wolfsbane is WARNING: DISK WARNING - free space: / 6260 MB (20% inode=93%): [01:04:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:18:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89371 MB (14% inode=99%): [01:21:49] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:21:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:25:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:43:27] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6225 MB (20% inode=93%): [01:58:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7751.000000 [01:58:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7757 [02:01:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:02:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:03:19] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:04:17] / on wolfsbane is WARNING: DISK WARNING - free space: / 6238 MB (20% inode=93%): [02:04:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:18:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89302 MB (14% inode=99%): [02:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:21:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:27] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6216 MB (20% inode=93%): [02:58:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5652.000000 [02:58:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5648 [03:01:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:02:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:03:19] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:04:18] / on wolfsbane is WARNING: DISK WARNING - free space: / 6225 MB (20% inode=93%): [03:04:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:08:28] /tmp on wolfsbane is OK: DISK OK - free space: / 6782 MB (22% inode=93%): [03:09:18] / on wolfsbane is OK: DISK OK - free space: / 6797 MB (22% inode=93%): [03:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89236 MB (14% inode=99%): [03:21:39] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:22:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:58:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3705.000000 [03:58:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3708 [04:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:01:48] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3544.000000 [04:01:48] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3543 [04:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:02:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:03:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:03:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:04:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89153 MB (14% inode=99%): [04:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:38:48] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1799.000000 [04:38:48] MySQL slave on rosemary is OK: Uptime: 207631 Threads: 14 Questions: 1473592720 Slow queries: 63894 Opens: 16759 Flush tables: 1 Open tables: 3008 Queries per second avg: 7097.171 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1800 [04:41:20] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.318 second response time [04:58:18] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:02:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:02:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:02:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:03:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:04:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89035 MB (14% inode=99%): [05:21:50] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:23:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:25:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:58:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:01:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:02:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:02:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:02:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:03:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:04:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88898 MB (14% inode=99%): [06:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:21:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:22:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:58:18] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:01:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:02:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:02:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:02:20] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:02:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:04:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:18:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88644 MB (14% inode=99%): [07:21:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:22:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:58:18] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:58:57] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199661 MB (20% inode=99%): [08:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:02:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:02:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:03:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:03:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:04:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:18:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88403 MB (14% inode=99%): [08:21:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:22:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:39:48] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:39:58] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:40:10] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:40:52] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2105.000000 [08:40:52] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2107 [08:40:52] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 31472 MB (7% inode=99%): [08:40:58] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 64728 MB (6% inode=97%): [08:40:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 64728 MB (6% inode=97%): [08:46:01] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:46:48] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:58:19] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:01:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:02:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:03:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:03:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:04:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:04:58] MySQL slave on z-dat-s3-a is OK: Uptime: 1472 Threads: 1 Questions: 5 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 10 Queries per second avg: 0.3 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [09:05:52] MySQL on z-dat-s3-a is OK: Uptime: 1527 Threads: 25 Questions: 570 Slow queries: 4 Opens: 254 Flush tables: 1 Open tables: 245 Queries per second avg: 0.373 [09:08:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3632.000000 [09:08:49] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3634 [09:16:58] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2191 [09:18:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88152 MB (14% inode=99%): [09:20:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 97570 MB (10% inode=99%): [09:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:21:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:38:29] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1967.000000 [09:42:58] MySQL slave on z-dat-s3-a is OK: Uptime: 3752 Threads: 14 Questions: 1145947 Slow queries: 138 Opens: 21010 Flush tables: 1 Open tables: 13184 Queries per second avg: 305.422 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1740 [09:58:30] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:03:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:04:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:08:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6769.000000 [10:08:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6770 [10:16:32] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1714.000000 [10:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87873 MB (14% inode=99%): [10:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:22:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:58:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:02:41] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:04:49] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:08:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4375.000000 [11:08:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4375 [11:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87657 MB (14% inode=99%): [11:21:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:22:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:26:48] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3540.000000 [11:26:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3539 [11:58:29] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:03:19] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:03:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:04:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87403 MB (14% inode=99%): [12:18:12] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 10.458 second response time [12:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:22:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:26:48] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2069.000000 [12:26:57] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2064 [12:38:50] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1779.000000 [12:38:57] MySQL slave on rosemary is OK: Uptime: 236433 Threads: 13 Questions: 1483850706 Slow queries: 73330 Opens: 18889 Flush tables: 1 Open tables: 3013 Queries per second avg: 6275.988 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1779 [12:41:18] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.957 second response time [12:50:58] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2101.000000 [12:50:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2101 [12:58:20] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.309 second response time [12:59:18] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 18.031 second response time [13:00:58] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1786.000000 [13:00:58] MySQL slave on rosemary is OK: Uptime: 237754 Threads: 10 Questions: 1484270577 Slow queries: 73919 Opens: 18921 Flush tables: 1 Open tables: 3013 Queries per second avg: 6242.883 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1785 [13:01:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:02:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:04:58] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:05:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:06:09] Marlen Caemmerer * [Toolserver-l] Maintenance of a Switch tomorrow [13:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87114 MB (14% inode=99%): [13:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:23:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:03:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:03:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:03:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:05:51] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86877 MB (14% inode=99%): [14:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:22:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:23:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:02:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:02:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:03:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:03:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:03:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:05:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86591 MB (14% inode=99%): [15:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:23:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:02:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:02:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:03:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:03:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:03:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:03:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:05:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86349 MB (14% inode=99%): [16:21:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:23:08] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:24:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:47:47] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:54:59] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1973 [16:55:07] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1974.000000 [17:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:02:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:03:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:03:19] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:03:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:03:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:05:57] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:11:09] Platonides * Re: [Toolserver-l] Maintenance of a Switch tomorrow [17:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86087 MB (14% inode=99%): [17:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:23:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:25:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:43:07] Platonides: did you get my messages? [17:54:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2647 [17:55:08] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2655.000000 [18:01:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:02:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:02:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:03:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:06:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:17:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85731 MB (14% inode=99%): [18:21:51] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:24:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:54:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3235 [18:55:08] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3237.000000 [19:01:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:02:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:02:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:03:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:03:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:03:27] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:03:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:06:58] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:10:02] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 57428 MB (5% inode=97%): [19:10:02] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 57404 MB (5% inode=97%): [19:18:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85504 MB (14% inode=99%): [19:21:52] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:23:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:40:58] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 64823 MB (6% inode=97%): [19:40:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 64827 MB (6% inode=97%): [19:52:14] nighty people o/ [19:54:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2993 [19:55:12] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2998.000000 [19:58:58] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199430 MB (20% inode=99%): [20:01:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:02:44] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:03:19] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:03:30] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:04:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:07:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85256 MB (14% inode=99%): [20:20:00] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3603 [20:20:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [20:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:23:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:41:03] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 31592 MB (7% inode=99%): [21:01:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:02:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:02:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:03:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:03:19] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:03:27] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:07:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84757 MB (13% inode=99%): [21:20:00] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4066 [21:20:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 93528 MB (9% inode=99%): [21:20:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4037.000000 [21:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:23:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:32:41] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3599.000000 [21:32:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3595 [21:46:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [21:47:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [22:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:02:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:02:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:04:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:04:27] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:07:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:18:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84506 MB (13% inode=99%): [22:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:22:38] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3582.000000 [22:22:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3561 [22:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:23:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:02:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:03:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:03:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:04:27] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:07:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:08:28] / on wolfsbane is WARNING: DISK WARNING - free space: / 6284 MB (20% inode=93%): [23:08:28] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6284 MB (20% inode=93%): [23:14:32] / on wolfsbane is OK: DISK OK - free space: / 6292 MB (21% inode=93%): [23:14:32] /tmp on wolfsbane is OK: DISK OK - free space: / 6292 MB (21% inode=93%): [23:18:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84295 MB (13% inode=99%): [23:21:31] / on wolfsbane is WARNING: DISK WARNING - free space: / 6283 MB (20% inode=93%): [23:21:31] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6283 MB (20% inode=93%): [23:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:23:02] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3147 [23:23:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:23:37] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [23:24:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:26:58] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [23:27:08] RAID on rosemary is CRITICAL: Connection refused by host [23:29:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [23:32:52] Load avg. on rosemary is CRITICAL: Connection refused by host [23:32:52] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [23:32:58] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [23:32:58] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [23:33:08] / on rosemary is CRITICAL: Connection refused by host [23:33:18] SMTP on rosemary is CRITICAL: Connection refused [23:33:28] SSH on rosemary is CRITICAL: Connection refused [23:33:28] /tmp on rosemary is CRITICAL: Connection refused by host [23:33:28] Environment IPMI on rosemary is CRITICAL: Connection refused by host