[00:04:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:28:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:41:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:43:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:46:33] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5758 [00:48:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5287.000000 [00:57:34] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3588 [00:58:14] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3536.000000 [01:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:14] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1701.000000 [01:17:34] MySQL slave on thyme is OK: Uptime: 18984 Threads: 5 Questions: 6861200 Slow queries: 2109 Opens: 345 Flush tables: 1 Open tables: 264 Queries per second avg: 361.420 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1640 [01:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:25:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:28:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:28:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:30:20] @replag [01:30:21] Brooke: s1-rr-a: 1m 32s [-2.15 s/s]; s1-user: 1m 19s [+0.00 s/s]; s2-user: 16s [-] [01:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:41:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:43:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:25:18] @replag all [02:25:18] Matthew_: s1-rr-a: 1s [-0.03 s/s]; s1-user: 1s [-0.02 s/s]; s2-user: 8s [-0.00 s/s]; s2-user-c: 1s [-]; s3-rr-a: 2m 7s [+0.00 s/s]; s3-user: 2m 7s [+0.00 s/s]; s4-rr-a: 1s [-]; s4-user: 1s [-] [02:25:19] Matthew_: s5-rr-a: 1s [-]; s5-user: 1s [-]; s5-user-c: 1s [-]; s6-rr-a: 3s [-]; s6-user: 3s [-]; s7-rr-a: 0s [-]; s7-user: 0s [-] [02:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:28:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:41:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:24:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:24:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:25:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:26:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:28:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:29:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:33:57] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:41:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:41:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:43:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:43:37] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:54:56] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70209 MB (7% inode=98%): [03:55:07] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 70208 MB (7% inode=98%): [04:04:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:04:57] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:19:36] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:24:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:24:36] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:57] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:25:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:26:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:28:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:29:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:33:56] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:41:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:41:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:43:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:43:37] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:04:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:24:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:24:36] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:57] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:25:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:26:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:28:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:29:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:33:57] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:41:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:41:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:43:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:43:36] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:04:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:24:16] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:24:36] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:57] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:25:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:26:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:28:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:29:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:33:57] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:39:57] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:41:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:41:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:43:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:43:36] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:44:37] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:04:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:24:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:24:36] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:57] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:25:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:26:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:28:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:29:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:33:57] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:41:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:41:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:43:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:43:37] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:04:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:24:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:24:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:25:16] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:26:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:29:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:32:37] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 34802 MB (8% inode=99%): [08:33:57] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:41:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:41:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:43:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:43:37] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:04:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:18:17] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 98873 MB (10% inode=99%): [09:24:17] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:24:36] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:56] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:25:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:26:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:29:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:33:57] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:34:47] Environment IPMI on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:36:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:36:21] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:43:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:05:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:21:49] connect: Lost connection to MySQL server at 'reading initial communication packet', system error: 0 [10:21:56] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:22:06] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:22:06] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:22:26] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35077 MB (8% inode=99%): [10:22:36] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 69871 MB (7% inode=98%): [10:22:36] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 69871 MB (7% inode=98%): [10:31:55] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:35:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:35:45] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:35:55] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:35:55] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:36:36] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:05:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:34:55] hello all [11:35:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:35:45] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:35:55] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:35:55] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:36:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:37:16] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2169 [11:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:37:56] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2203.000000 [11:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:42:29] Hi DaBPunkt [11:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:01:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [12:02:15] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3643 [12:05:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:14:48] @replag [12:14:49] DaBPunkt: s1-rr-a: 1h 12m 40s [+0.12 s/s] [12:24:55] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3526.000000 [12:25:15] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3434 [12:35:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:35:45] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:37:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:37:56] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1724.000000 [12:38:16] MySQL slave on thyme is OK: Uptime: 59834 Threads: 3 Questions: 16280169 Slow queries: 6588 Opens: 615 Flush tables: 1 Open tables: 336 Queries per second avg: 272.88 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1615 [12:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:59:36] /sql on thyme is WARNING: DISK WARNING - free space: /sql 193951 MB (20% inode=99%): [13:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:23:36] /sql on thyme is OK: DISK OK - free space: /sql 202484 MB (21% inode=99%): [13:32:36] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198602 MB (20% inode=99%): [13:34:36] /sql on thyme is OK: DISK OK - free space: /sql 203518 MB (21% inode=99%): [13:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:37:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:40:16] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2159 [13:40:55] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2193.000000 [13:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:44:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:44:21] According to MUNIN thyme's been pegged by a busy disk for 2 days [13:46:48] With a corresponding uptick in memory usage [13:53:56] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1731.000000 [13:54:15] MySQL slave on thyme is OK: Uptime: 64393 Threads: 9 Questions: 16868484 Slow queries: 8365 Opens: 652 Flush tables: 1 Open tables: 371 Queries per second avg: 261.961 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1711 [14:02:13] DaBPunkt: Could you check thyme for 10 rouge processes taking up 5 GB of memory started Oct 13 around 18:00 UTC and hammering the disk? [14:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:06:54] Dispenser: How should I do this? [14:07:01] ps maybe? [14:08:04] Dispenser: thyme is a database-server [14:26:56] which is why I find a spike in # of processes and memory to be suspicious [14:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:31:28] @replag [15:31:32] DaBPunkt: s1-rr-a: 2m 16s [-0.36 s/s]; s1-user: 2m 5s [+0.00 s/s] [15:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:36:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:44:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:36:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:36:25] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:41:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:43:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:36:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:41:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:43:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:35:55] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:35:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:41:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:43:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:05:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:35:55] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:35:55] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:36:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:18:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 98479 MB (10% inode=99%): [21:35:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:35:45] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:36:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:36:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:05:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:22:26] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 33702 MB (8% inode=99%): [22:22:36] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70141 MB (7% inode=98%): [22:22:36] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 70141 MB (7% inode=98%): [22:35:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:37:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:43:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:51:05] nacht ts [23:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:35:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:35:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:35:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:35:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:36:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:36:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:41:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:41:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:43:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:44:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: