[00:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:10:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28539.000000 [00:13:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 6274 MB (20% inode=93%): [00:16:50] / on wolfsbane is OK: DISK OK - free space: / 6299 MB (21% inode=93%): [00:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77925 MB (12% inode=99%): [00:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:23:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 6273 MB (20% inode=93%): [00:24:00] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6273 MB (20% inode=93%): [00:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:25:01] /tmp on wolfsbane is OK: DISK OK - free space: / 6296 MB (21% inode=93%): [00:37:01] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6238 MB (20% inode=93%): [00:41:30] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5493 [00:41:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5484.000000 [01:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:10:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30904.000000 [01:18:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 6274 MB (20% inode=93%): [01:20:00] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6256 MB (20% inode=93%): [01:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77828 MB (12% inode=99%): [01:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:41:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4426 [01:41:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4437.000000 [02:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:10:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34039.000000 [02:17:30] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3599 [02:17:51] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3549.000000 [02:18:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 6208 MB (20% inode=93%): [02:20:01] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6213 MB (20% inode=93%): [02:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77785 MB (12% inode=99%): [02:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:24:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [02:25:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3617 [02:25:50] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3599.000000 [02:26:30] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3560 [02:59:50] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1755.000000 [03:00:31] MySQL slave on rosemary is OK: Uptime: 15334983 Threads: 17 Questions: 6797055075 Slow queries: 2198540 Opens: 311677 Flush tables: 6 Open tables: 4155 Queries per second avg: 443.238 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1645 [03:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:07:51] / on wolfsbane is OK: DISK OK - free space: / 7358 MB (24% inode=93%): [03:08:00] /tmp on wolfsbane is OK: DISK OK - free space: / 7349 MB (24% inode=93%): [03:11:40] s4 replag on cassia is CRITICAL: (Service Check Timed Out) [03:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77732 MB (12% inode=99%): [03:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:12:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41027.000000 [04:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77650 MB (12% inode=99%): [04:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:49:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 4808 MB (16% inode=93%): [04:51:01] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4796 MB (16% inode=93%): [04:57:51] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2174.000000 [04:58:11] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2201 [04:59:51] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:12:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43657.000000 [05:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3611.000000 [05:22:11] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3640 [05:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77585 MB (12% inode=99%): [05:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:50:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4766 MB (15% inode=93%): [05:51:01] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4766 MB (15% inode=93%): [06:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:12:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47229.000000 [06:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7209.000000 [06:22:11] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7236 [06:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77522 MB (12% inode=99%): [06:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:46:08] @replag [06:46:09] liangent: s2-user: 1d 17h 27m 58s [+0.93 s/s]; s2-user-c: 13h 41m 6s [+0.91 s/s]; s5-user-c: 13h 41m 7s [+0.91 s/s] [06:50:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4734 MB (15% inode=93%): [06:51:00] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4722 MB (15% inode=93%): [07:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:01:50] / on wolfsbane is OK: DISK OK - free space: / 14249 MB (47% inode=93%): [07:02:00] /tmp on wolfsbane is OK: DISK OK - free space: / 14249 MB (47% inode=93%): [07:12:30] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50838.000000 [07:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10806.000000 [07:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10834 [07:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77453 MB (12% inode=99%): [07:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:12:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54099.000000 [08:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14404.000000 [08:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14431 [08:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77356 MB (12% inode=99%): [08:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:12:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57699.000000 [09:17:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 103295 MB (10% inode=99%): [09:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17998.000000 [09:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18027 [09:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77257 MB (12% inode=99%): [09:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:54:51] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:55:01] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:55:01] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:56:51] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73441 MB (7% inode=98%): [09:56:51] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73441 MB (7% inode=98%): [09:56:51] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37646 MB (9% inode=99%): [10:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:12:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59496.000000 [10:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21597.000000 [10:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21625 [10:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77107 MB (12% inode=99%): [10:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:12:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60127.000000 [11:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25193.000000 [11:22:11] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25222 [11:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76925 MB (12% inode=99%): [11:23:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:23:42] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:12:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60718.000000 [12:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28794.000000 [12:22:11] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28820 [12:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76770 MB (12% inode=99%): [12:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:24:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:26:51] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:01] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:01] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:21] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37443 MB (9% inode=99%): [12:27:31] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73291 MB (7% inode=98%): [12:27:31] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73291 MB (7% inode=98%): [12:44:44] hello all [13:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:12:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62411.000000 [13:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32394.000000 [13:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32419 [13:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76602 MB (12% inode=99%): [13:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:24:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:12:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64393.000000 [14:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35990.000000 [14:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36018 [14:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76389 MB (12% inode=99%): [14:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:24:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:24:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:07:33] DaBPunkt: can you please install the package tclthread on a linux login server? [15:07:42] giftpflanze: is there a jira-bug? [15:07:52] i can make you one [15:07:57] yes please [15:08:00] ok [15:10:08] DaBPunkt: TS-1536 [15:10:50] ok. I will install new packages somewhen tonight (22 o'clock UTC). Is that afst enough? [15:11:02] UTC ≥ MESZ [15:11:46] okay [15:12:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66369.000000 [15:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39591.000000 [15:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39616 [15:23:00] DaBPunkt: Is that battery problem related with the recent storage issues? [15:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76187 MB (12% inode=99%): [15:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:51] multichill: no, that's the local array AFAIS [15:33:55] Hmm, hardware is falling apart. I didn't dare to read the whole future thread..... [15:53:26] !replag [15:53:41] @replag [15:53:42] akoopal: s1-rr-a: 5m 45s [-0.05 s/s]; s1-user: 5m 45s [-0.05 s/s]; s2-user: 2d 1h 24s [+0.77 s/s]; s2-user-c: 18h 38m 31s [+0.34 s/s]; s3-rr-a: 1m 2s [+0.01 s/s]; s3-user: 1m 3s [+0.01 s/s]; s5-user-c: 18h 38m 32s [+0.34 s/s]; s6-rr-a: 36s [-0.00 s/s] [15:53:43] akoopal: s6-user: 36s [-0.00 s/s] [15:54:12] s2 is going bad again :-( [16:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:12:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66692.000000 [16:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43186.000000 [16:22:11] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43214 [16:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76013 MB (12% inode=99%): [16:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:12:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67986.000000 [17:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46786.000000 [17:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46813 [17:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75683 MB (12% inode=99%): [17:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:24:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:02:00] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 184.00 ms [18:03:11] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 186.00 ms [18:05:01] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 140.00 ms [18:11:10] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 156.00 ms [18:12:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68743.000000 [18:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50384.000000 [18:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50412 [18:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75437 MB (12% inode=99%): [18:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:54:11] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 164.00 ms [18:56:10] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 151.00 ms [18:57:14] re [18:57:21] So, the rest of the day is for the ts [19:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:12:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69685.000000 [19:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53984.000000 [19:22:11] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54011 [19:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75192 MB (12% inode=99%): [19:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:24:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:12:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70362.000000 [20:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57583.000000 [20:22:11] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57610 [20:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74979 MB (12% inode=99%): [20:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:25:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:40:54] @replag [20:40:54] akoopal: s1-rr-a: 10s [-0.02 s/s]; s1-user: 10s [-0.02 s/s]; s2-user: 2d 4h 55m 42s [+0.84 s/s]; s2-user-c: 19h 22m 16s [+0.13 s/s]; s5-user-c: 19h 22m 16s [+0.13 s/s] [20:41:15] DaBPunkt: any idea what is happening to s2-user? [20:42:57] akoopal: the server is busy. The replication is running AFAIS [20:43:40] it seems to be, yes, but not keeping up at all :-( [20:45:14] akoopal: the problem here is that the query-killer can only measure the replag of the mysql-replication-process – not of the trainwrecks [20:45:52] what do you mean with trainwrecks? [20:46:28] akoopal: trainwreck is the software we use to replicate databases [20:46:35] ahh [20:48:14] So it is just getting to many query's if I understand correctly [20:50:38] akoopal: more or less: yes [20:51:17] ok, nothing to do about it then sort term :-( [21:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:12:30] @replag [21:12:30] DaBPunkt: s1-rr-a: 11s [-0.21 s/s]; s1-user: 11s [-0.21 s/s]; s2-user: 2d 5h 24m 12s [+0.87 s/s]; s2-user-c: 19h 20m 9s [+0.82 s/s]; s5-user-c: 19h 20m 9s [+0.81 s/s] [21:12:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69609.000000 [21:16:03] @replag [21:16:04] DaBPunkt: s1-rr-a: 32s [+0.10 s/s]; s1-user: 32s [+0.10 s/s]; s2-user: 2d 5h 27m 27s [+0.91 s/s]; s2-user-c: 19h 21m 7s [+0.27 s/s]; s3-rr-a: 45s [+0.08 s/s]; s3-user: 45s [+0.08 s/s]; s5-user-c: 19h 21m 7s [+0.27 s/s] [21:17:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 103139 MB (10% inode=99%): [21:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61184.000000 [21:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61208 [21:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74696 MB (12% inode=99%): [21:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:25:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:12:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63996.000000 [22:21:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64780.000000 [22:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64805 [22:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74442 MB (12% inode=99%): [22:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:24:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:25:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:12:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49192.000000 [23:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68377.000000 [23:22:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 68403 [23:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74269 MB (12% inode=99%): [23:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:24:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:25:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default