[00:10:34] Free Memory on damiana is WARNING: WARNING - 6.8% (565824 kB) free! [00:11:43] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [00:16:34] Free Memory on damiana is OK: OK - 7.2% (604716 kB) free. [00:18:46] @replag [00:18:47] Brooke: s1-rr-a: 6m 14s [-0.08 s/s]; s1-user: 6m 14s [-0.08 s/s]; s2-user: 13s [-0.00 s/s]; s3-rr-a: 35s [+0.00 s/s]; s3-user: 35s [+0.00 s/s] [00:18:52] Oh, six minutes. [00:18:54] That explains that. [00:19:04] Brooke: Stop assuming lag is 0. [00:24:03] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:33:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:34:04] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [00:36:05] @replag [00:36:05] Brooke: s1-rr-a: 13m 46s [+0.44 s/s]; s1-user: 13m 46s [+0.44 s/s]; s3-rr-a: 7m 12s [+0.38 s/s]; s3-user: 7m 12s [+0.38 s/s] [00:36:10] Huh. [00:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:44:12] Brooke: that's more or less normal, because most data analyse tools running complex queries run after midnight on ts [00:44:35] Right. [00:44:44] 0 * * * * or whatever. [00:45:40] if you want to collect data of a hole day it make sense to wait unil the day is over [00:58:44] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [01:05:34] Free Memory on damiana is WARNING: WARNING - 5.5% (460044 kB) free! [01:11:43] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [01:16:33] Free Memory on damiana is OK: OK - 7.1% (595996 kB) free. [01:24:03] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:33:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:34:04] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [01:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:58:44] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [02:11:43] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [02:24:03] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:30:34] Free Memory on damiana is WARNING: WARNING - 5.1% (425824 kB) free! [02:33:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:34:04] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [02:38:34] Free Memory on damiana is OK: OK - 7.2% (599372 kB) free. [02:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:55:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71815 MB (7% inode=98%): [02:55:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71815 MB (7% inode=98%): [02:56:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35514 MB (8% inode=99%): [02:58:44] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [03:11:43] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [03:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:33:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:34:04] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [03:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:58:44] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [04:05:14] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:44] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71722 MB (7% inode=98%): [04:11:45] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [04:24:03] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:34:04] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [04:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:53:24] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:55:23] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71551 MB (7% inode=98%): [04:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [05:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [05:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [05:34:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [05:59:24] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [06:11:34] Free Memory on damiana is WARNING: WARNING - 6.2% (516452 kB) free! [06:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [06:23:33] Free Memory on damiana is CRITICAL: CRITICAL - 3.6% (305628 kB) free! [06:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:34] Free Memory on damiana is OK: OK - 9.7% (811664 kB) free. [06:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [06:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [07:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [07:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:26:22] well, typing obama's net worth into vanilla google gives you the answer, in big bold letters [07:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [07:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [08:01:34] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419756 kB) free! [08:04:34] Free Memory on damiana is WARNING: WARNING - 5.1% (429508 kB) free! [08:10:34] Free Memory on damiana is OK: OK - 7.4% (623640 kB) free. [08:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [08:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:32:33] Free Memory on damiana is WARNING: WARNING - 6.0% (505632 kB) free! [08:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [08:40:34] Free Memory on damiana is OK: OK - 7.1% (592256 kB) free. [08:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [09:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [09:18:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 101960 MB (10% inode=99%): [09:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:30:33] Free Memory on damiana is WARNING: WARNING - 5.6% (466548 kB) free! [09:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [09:34:34] Free Memory on damiana is OK: OK - 7.3% (611128 kB) free. [09:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [10:08:34] Free Memory on damiana is WARNING: WARNING - 5.8% (489452 kB) free! [10:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [10:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [11:03:34] Free Memory on damiana is WARNING: WARNING - 5.1% (430612 kB) free! [11:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [11:17:34] Free Memory on damiana is OK: OK - 7.2% (601876 kB) free. [11:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [11:41:34] Free Memory on damiana is WARNING: WARNING - 6.3% (525256 kB) free! [11:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:47:31] any admin with tomcat knowledge available who could solve ticket TS-1550, please? [11:55:34] Free Memory on damiana is OK: OK - 7.1% (592184 kB) free. [11:58:04] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 75.77, 59.16, 30.92 [11:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [11:59:54] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 6.533 second response time [12:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [12:18:44] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.680 second response time [12:19:43] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.985 second response time [12:21:43] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.423 second response time [12:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [12:40:57] problems in ortelius, returns a 502 [12:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:44:14] this is odd, the same script works in wolfsbane but fails in ortelius [12:44:20] yet other pages work in both [12:58:04] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 76.65, 73.85, 67.87 [12:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [13:11:34] Free Memory on damiana is WARNING: WARNING - 6.3% (525572 kB) free! [13:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [13:23:34] Free Memory on damiana is OK: OK - 7.1% (595136 kB) free. [13:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:35] hello all [13:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:27:03] Load avg. on ortelius is WARNING: WARNING - load average: 2.84, 3.60, 19.44 [13:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:32:04] Load avg. on ortelius is OK: OK - load average: 2.16, 2.75, 14.61 [13:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [13:42:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:58:34] Hi DaBPunkt. Filed a ticket according to my request regarding postgis-jars here: https://jira.toolserver.org/browse/TS-1550 [13:58:51] thnx [13:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [13:59:39] should I assign it to anybody? [13:59:54] no thans [14:01:13] k, thanks ;) [14:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [14:19:34] Free Memory on damiana is WARNING: WARNING - 5.7% (475068 kB) free! [14:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:27:34] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (402584 kB) free! [14:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:29:34] Free Memory on damiana is WARNING: WARNING - 5.7% (474544 kB) free! [14:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [14:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:55:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 69669 MB (7% inode=98%): [14:56:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35323 MB (8% inode=99%): [14:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [15:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [15:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [15:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [16:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [16:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [16:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:54:33] Free Memory on damiana is WARNING: WARNING - 6.5% (547000 kB) free! [16:55:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 69352 MB (7% inode=98%): [16:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [17:08:33] Free Memory on damiana is OK: OK - 7.4% (619296 kB) free. [17:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [17:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [17:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [18:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [18:16:34] Free Memory on damiana is WARNING: WARNING - 6.1% (512540 kB) free! [18:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:28:33] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417264 kB) free! [18:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:29:34] Free Memory on damiana is WARNING: WARNING - 6.3% (529520 kB) free! [18:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [18:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:52:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:56:24] Environment IPMI on wolfsbane is OK: ok: temperature ok fan ok voltage ok chassis ok [18:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [19:04:36] hello [19:04:48] i will shut down sql-s1-user quite soon [19:04:53] its maintenance window [19:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [19:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:15:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [19:15:44] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [19:15:44] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [19:15:44] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [19:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:31:34] Free Memory on damiana is WARNING: WARNING - 6.2% (516500 kB) free! [19:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [19:38:34] Free Memory on damiana is OK: OK - 7.1% (592780 kB) free. [19:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [20:00:34] Free Memory on damiana is WARNING: WARNING - 5.8% (487080 kB) free! [20:03:43] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3555.000000 [20:03:43] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3256.000000 [20:03:43] MySQL on rosemary is OK: Uptime: 67 Threads: 25 Questions: 8378 Slow queries: 61 Opens: 280 Flush tables: 1 Open tables: 273 Queries per second avg: 125.44 [20:03:44] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3558 [20:04:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [20:04:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3603 [20:07:17] maintenance finished [20:10:34] Free Memory on damiana is OK: OK - 7.1% (598372 kB) free. [20:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [20:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:21:54] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [20:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [20:39:44] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3507 [20:39:54] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3501.000000 [20:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:46:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3626 [20:46:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [20:54:44] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3559 [20:54:54] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3555.000000 [20:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [21:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [21:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:18:04] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 99607 MB (10% inode=99%): [21:19:06] Sumana Harihareswara * [Toolserver-l] Fwd: Please notice and report glitches - changes coming [21:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:27:44] MySQL slave on rosemary is OK: Uptime: 5108 Threads: 10 Questions: 2422898 Slow queries: 1596 Opens: 1089 Flush tables: 1 Open tables: 463 Queries per second avg: 474.333 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [21:27:54] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1773.000000 [21:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:33:34] Free Memory on damiana is WARNING: WARNING - 6.6% (557156 kB) free! [21:34:13] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [21:34:34] Free Memory on damiana is OK: OK - 7.2% (605696 kB) free. [21:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:45:34] Free Memory on damiana is WARNING: WARNING - 6.3% (527612 kB) free! [21:58:53] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [22:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [22:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:33:34] Free Memory on damiana is WARNING: WARNING - 6.2% (521376 kB) free! [22:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [22:37:33] Free Memory on damiana is OK: OK - 7.1% (591084 kB) free. [22:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [23:11:54] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [23:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:49] nacht ts [23:24:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:34:14] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [23:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:58:54] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat