[00:06:34] Free Memory on damiana is WARNING: WARNING - 5.7% (478112 kB) free! [00:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [00:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:28:33] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (402752 kB) free! [00:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:29:34] Free Memory on damiana is WARNING: WARNING - 6.1% (510472 kB) free! [00:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [00:36:34] Free Memory on damiana is OK: OK - 7.3% (609764 kB) free. [00:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [01:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [01:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [01:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [02:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [02:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [02:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:55:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70927 MB (7% inode=98%): [02:56:33] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35431 MB (8% inode=99%): [02:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [03:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [03:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [03:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:44:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [04:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [04:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:14:34] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [04:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:46:14] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:24] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:03] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:44] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35319 MB (8% inode=99%): [04:47:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71667 MB (7% inode=98%): [04:47:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71667 MB (7% inode=98%): [04:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [05:11:53] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [05:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:24:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [05:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [06:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [06:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:27:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [06:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [07:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [07:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [07:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [07:59:03] Load avg. on adenia is WARNING: WARNING - load average: 17.07, 14.32, 8.66 [08:11:04] Load avg. on adenia is OK: OK - load average: 12.84, 14.85, 12.36 [08:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [08:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [08:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:58:55] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [09:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [09:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:18:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 99641 MB (10% inode=99%): [09:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [09:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [10:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [10:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [10:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:39:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:44:24] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:47:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71434 MB (7% inode=98%): [16:47:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71434 MB (7% inode=98%): [16:48:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35397 MB (8% inode=99%): [17:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:24:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:32:04] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:32:14] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:32:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:32:33] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 34511 MB (8% inode=99%): [20:32:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71094 MB (7% inode=98%): [20:32:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71092 MB (7% inode=98%): [20:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:18:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 99423 MB (10% inode=99%): [21:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:56:27] @replag [21:56:27] Brooke: s1-rr-a: 4m 17s [-0.01 s/s]; s1-user: 4m 17s [-0.01 s/s]; s2-user: 13s [-]; s3-rr-a: 28s [+0.00 s/s]; s3-user: 28s [+0.00 s/s] [22:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:24:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: