[00:16:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:17:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [00:18:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:19:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:20:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:22:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:22:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:23:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:25:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73451 MB (12% inode=99%): [00:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:36:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:38:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:17:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [01:18:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:19:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:20:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:22:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:22:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:23:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:25:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73363 MB (12% inode=99%): [01:35:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:36:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:38:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:46:15] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106944 MB (11% inode=99%): [01:47:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106404 MB (10% inode=99%): [02:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:17:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [02:18:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:19:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:20:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:22:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:22:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:23:45] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:25:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73262 MB (12% inode=99%): [02:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:36:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:36:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:38:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:44:15] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106969 MB (11% inode=99%): [02:45:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106416 MB (10% inode=99%): [03:17:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:18:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [03:19:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:19:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:21:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:23:45] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:23:45] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:24:45] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:26:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:34:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73162 MB (12% inode=99%): [03:36:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:37:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:37:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:39:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:18:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:18:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:19:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:19:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:21:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:23:45] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:23:45] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:24:45] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:26:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:34:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73105 MB (11% inode=99%): [04:36:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:37:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:37:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:39:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:01:46] Environment IPMI on wolfsbane is WARNING: NRPE: Unable to read output [05:03:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:18:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:18:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:19:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:19:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:21:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:23:55] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:23:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:33:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:33:36] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107415 MB (11% inode=99%): [05:34:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72997 MB (11% inode=99%): [05:36:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:37:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:37:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:39:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:04:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:18:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:18:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:19:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:20:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:20:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106926 MB (10% inode=99%): [06:21:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:23:45] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:23:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:24:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs