[00:00:36] Free Memory on damiana is WARNING: WARNING - 5.5% (457688 kB) free! [00:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:10:37] Free Memory on damiana is OK: OK - 7.2% (605632 kB) free. [00:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111696 MB (18% inode=99%): [00:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:26:39] Free Memory on damiana is WARNING: WARNING - 5.6% (465284 kB) free! [00:27:41] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (366804 kB) free! [00:34:05] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:36:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:36:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:38:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14123 [00:42:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14159.000000 [00:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:58:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201457 MB (20% inode=99%): [01:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:15:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111582 MB (18% inode=99%): [01:16:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:34:05] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:37:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:42:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15279 [01:42:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15273.000000 [01:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111521 MB (18% inode=99%): [02:16:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:34:07] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:37:09] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:37:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15770 [02:42:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15780.000000 [02:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111460 MB (18% inode=99%): [03:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:34:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:36:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:38:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18595 [03:43:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18633.000000 [03:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111378 MB (18% inode=99%): [04:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:36:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:37:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:38:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19085 [04:43:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19087.000000 [04:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:55:35] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419896 kB) free! [04:56:36] Free Memory on damiana is WARNING: WARNING - 5.3% (444580 kB) free! [05:04:35] Free Memory on damiana is OK: OK - 9.6% (804632 kB) free. [05:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:15:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111240 MB (18% inode=99%): [05:16:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:36:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:36:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:37:19] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:42:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19717 [05:43:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19763.000000 [05:44:28] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:16:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111093 MB (18% inode=99%): [06:16:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:31:27] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:31:27] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [06:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:37:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:37:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:38:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:42:09] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22469 [06:43:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22504.000000 [06:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:05:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:16:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110851 MB (18% inode=99%): [07:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [07:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [07:32:26] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [07:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:36:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:37:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:42:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24646 [07:43:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24671.000000 [07:44:34] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:16:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110627 MB (18% inode=99%): [08:16:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:32:27] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:32:27] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [08:34:07] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:37:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26383 [08:43:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26433.000000 [08:44:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:54:26] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2170 [08:55:15] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [09:06:00] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:16:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110380 MB (18% inode=99%): [09:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3601 [09:19:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3646.000000 [09:19:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 94764 MB (9% inode=99%): [09:23:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:32:26] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:38:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29550 [09:43:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29634.000000 [09:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:05:59] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:16:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110065 MB (18% inode=99%): [10:16:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7134 [10:19:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7176.000000 [10:22:29] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 31367 MB (7% inode=99%): [10:23:06] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 64610 MB (6% inode=97%): [10:23:06] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 64610 MB (6% inode=97%): [10:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:26:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:32:29] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:32:29] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:34:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:37:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:37:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:37:29] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:42:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31522 [10:43:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31504.000000 [10:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:08:36] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:09:25] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 31340 MB (7% inode=99%): [11:15:06] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [11:15:06] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [11:15:06] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [11:15:06] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [11:16:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109808 MB (18% inode=99%): [11:16:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:17:06] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1953 [11:17:06] MySQL on z-dat-s6-a is OK: Uptime: 505 Threads: 3 Questions: 6580 Slow queries: 0 Opens: 108 Flush tables: 1 Open tables: 98 Queries per second avg: 13.29 [11:18:05] MySQL slave on z-dat-s6-a is OK: Uptime: 564 Threads: 3 Questions: 29087 Slow queries: 1 Opens: 128 Flush tables: 1 Open tables: 117 Queries per second avg: 51.572 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1736 [11:18:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10628 [11:20:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10700.000000 [11:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:31:06] MySQL on z-dat-s3-a is OK: Uptime: 1344 Threads: 9 Questions: 6705 Slow queries: 1 Opens: 246 Flush tables: 1 Open tables: 225 Queries per second avg: 4.988 [11:31:07] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3123 [11:31:27] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:32:26] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:36:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:37:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:37:07] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32075 [11:42:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3622 [11:43:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32023.000000 [11:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:51:06] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3475 [11:59:06] MySQL slave on z-dat-s3-a is OK: Uptime: 3025 Threads: 14 Questions: 441874 Slow queries: 29 Opens: 9929 Flush tables: 1 Open tables: 9906 Queries per second avg: 146.74 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1573 [12:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:16:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109530 MB (18% inode=99%): [12:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:18:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14059 [12:20:26] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [12:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:32:29] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:32:29] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:34:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:36:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:37:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:38:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33095 [12:43:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33176.000000 [12:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:58:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200514 MB (20% inode=99%): [13:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109244 MB (17% inode=99%): [13:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:18:30] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17311 [13:20:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17446.000000 [13:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:26:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:27:56] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1843.000000 [13:30:55] s4 replag on rosemary is CRITICAL: (Service Check Timed Out) [13:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:32:30] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:32:30] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:34:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:34:39] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2115.000000 [13:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:37:20] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34236 [13:44:27] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34219.000000 [13:44:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:52:53] @replag [13:52:53] legoktm: s1-rr-a: 5h 21m 39s [+0.08 s/s]; s1-user: 9h 31m 19s [+0.35 s/s]; s2-user-c: 31s [-0.08 s/s]; s3-rr-a: 1m 43s [+0.00 s/s]; s3-user: 1m 43s [+0.00 s/s]; s5-user-c: 31s [-0.08 s/s]; s7-rr-a: 10s [-0.01 s/s]; s7-user: 10s [-0.01 s/s] [14:01:36] s4 replag on rosemary is CRITICAL: (Service Check Timed Out) [14:01:55] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2670.000000 [14:05:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:15:08] Alana Kingston * Re: [Toolserver-l] Mail forwarding not working [14:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108956 MB (17% inode=99%): [14:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20780 [14:20:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20925.000000 [14:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:26:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:31:28] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [14:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [14:32:27] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [14:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:36:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:37:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:38:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36405 [14:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:44:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36531.000000 [15:02:11] @replag [15:02:14] Dispenser: s1-rr-a: 6h 29m 45s [+0.98 s/s]; s1-user: 10h 23m 26s [+0.75 s/s]; s2-user: 13s [-0.06 s/s]; s3-rr-a: 1m 10s [-0.01 s/s]; s3-user: 1m 10s [-0.01 s/s]; s6-rr-a: 13s [-0.00 s/s]; s6-user: 14s [-0.00 s/s]; s7-rr-a: 55s [+0.01 s/s] [15:02:15] Dispenser: s7-user: 55s [+0.01 s/s] [15:02:46] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2656.000000 [15:03:31] Looks like our little disk trashing bug's come back [15:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108617 MB (17% inode=99%): [15:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24335 [15:21:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24479.000000 [15:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:31:29] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [15:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [15:32:26] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [15:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:36:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:37:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:37:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:42:09] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37082 [15:42:09] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2115 [15:44:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:44:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37100.000000 [15:46:08] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2195 [15:48:06] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2162 [16:02:45] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1976.000000 [16:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:10:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3626 [16:12:07] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3643 [16:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108291 MB (17% inode=99%): [16:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:18:28] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27719 [16:19:07] MySQL slave on z-dat-s6-a is OK: Uptime: 18624 Threads: 15 Questions: 2256744 Slow queries: 5170 Opens: 1011 Flush tables: 1 Open tables: 489 Queries per second avg: 121.173 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1737 [16:19:46] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [16:21:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27913.000000 [16:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:31:27] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [16:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [16:32:26] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [16:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:36:40] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:37:08] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:41:08] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3567 [16:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39003 [16:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:44:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38878.000000 [16:47:06] MySQL slave on z-dat-s7-a is OK: Uptime: 20304 Threads: 20 Questions: 8975331 Slow queries: 2215 Opens: 94787 Flush tables: 1 Open tables: 929 Queries per second avg: 442.47 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1730 [16:48:06] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3556 [16:56:06] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [17:02:10] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [17:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:15:07] MySQL on z-dat-s3-a is OK: Uptime: 1141 Threads: 14 Questions: 3754 Slow queries: 0 Opens: 149 Flush tables: 1 Open tables: 131 Queries per second avg: 3.290 [17:15:07] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3017 [17:16:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108056 MB (17% inode=99%): [17:16:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31191 [17:22:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31392.000000 [17:23:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:27:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3645 [17:30:10] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3520 [17:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [17:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [17:32:26] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [17:34:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:36:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:37:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:37:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:40:06] MySQL slave on z-dat-s3-a is OK: Uptime: 2642 Threads: 15 Questions: 300271 Slow queries: 23 Opens: 14390 Flush tables: 1 Open tables: 8726 Queries per second avg: 113.652 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1631 [17:42:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38079 [17:44:28] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:44:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37990.000000 [18:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107769 MB (17% inode=99%): [18:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34727 [18:23:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34951.000000 [18:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [18:32:29] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [18:32:29] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [18:34:07] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:36:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:37:11] DaB. * Re: [Toolserver-l] Mail forwarding not working [18:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:38:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37706 [18:44:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:44:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37687.000000 [19:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:14:06] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2442 MB (16% inode=99%): [19:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107482 MB (17% inode=99%): [19:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:18:27] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38252 [19:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:23:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38518.000000 [19:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:29:06] /tmp on ortelius is OK: DISK OK - free space: /tmp 3282 MB (21% inode=99%): [19:31:27] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [19:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [19:32:26] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [19:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:37:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:37:09] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:37:09] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2160 [19:37:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:38:06] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2133 [19:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:41:05] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 977 MB (6% inode=99%): [19:41:06] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2109 [19:42:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32497 [19:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:44:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32423.000000 [20:03:55] RAID on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:04:07] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3629 [20:04:36] RAID on rosemary is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:05:59] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:07:12] !replag [20:09:05] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3602 [20:09:06] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3639 [20:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107178 MB (17% inode=99%): [20:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41781 [20:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:24:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42122.000000 [20:25:17] @replag [20:25:52] legoktm[NE]: s1-rr-a: 11h 43m 13s [+0.97 s/s]; s1-user: 8h 33m 38s [-0.34 s/s]; s3-rr-a: 1h 15m 40s [+0.23 s/s]; s3-user: 1h 15m 45s [+0.23 s/s]; s6-rr-a: 1h 12m 38s [+0.22 s/s]; s6-user: 1h 12m 38s [+0.22 s/s]; s7-rr-a: 1h 21m 52s [+0.25 s/s]; s7-user: 1h 21m 52s [+0.25 s/s] [20:26:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [20:32:27] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [20:32:27] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [20:34:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:37:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:38:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:40:56] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 12.286 second response time [20:41:10] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1204 MB (7% inode=99%): [20:42:08] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29830 [20:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:45:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29831.000000 [21:04:06] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5667 [21:05:59] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:28] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [21:09:08] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4089 [21:09:08] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5119 [21:10:26] MySQL on thyme is OK: Uptime: 336 Threads: 4 Questions: 5811 Slow queries: 1 Opens: 75 Flush tables: 1 Open tables: 68 Queries per second avg: 17.294 [21:11:06] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3285 [21:14:09] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3578 [21:15:06] MySQL slave on z-dat-s7-a is OK: Uptime: 36384 Threads: 5 Questions: 10377281 Slow queries: 5653 Opens: 94991 Flush tables: 1 Open tables: 1062 Queries per second avg: 285.215 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1748 [21:16:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106862 MB (17% inode=99%): [21:16:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:18:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43518 [21:19:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 95031 MB (9% inode=99%): [21:23:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:25:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41737.000000 [21:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:29:06] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3549 [21:31:27] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [21:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [21:33:26] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [21:36:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:36:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:41:09] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1218 MB (7% inode=99%): [21:42:16] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29909 [21:43:06] MySQL slave on z-dat-s6-a is OK: Uptime: 38064 Threads: 20 Questions: 4111644 Slow queries: 7728 Opens: 9073 Flush tables: 1 Open tables: 542 Queries per second avg: 108.19 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1755 [21:44:06] MySQL slave on z-dat-s3-a is OK: Uptime: 17282 Threads: 27 Questions: 12513504 Slow queries: 4377 Opens: 205472 Flush tables: 1 Open tables: 16384 Queries per second avg: 724.77 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1743 [21:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:45:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29976.000000 [22:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106565 MB (17% inode=99%): [22:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29750 [22:23:08] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 64638 MB (6% inode=97%): [22:23:08] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 64638 MB (6% inode=97%): [22:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:25:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29409.000000 [22:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:31:30] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [22:32:28] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [22:33:26] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [22:36:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:37:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:38:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:41:06] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1252 MB (7% inode=99%): [22:42:18] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29205 [22:44:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:45:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29205.000000 [23:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:09:26] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 31211 MB (7% inode=99%): [23:16:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106241 MB (17% inode=99%): [23:16:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:18:30] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23957 [23:23:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:25:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23031.000000 [23:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:31:28] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [23:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [23:33:26] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [23:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:37:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:37:10] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:37:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:41:06] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1477 MB (9% inode=99%): [23:42:16] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29154 [23:44:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:45:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29165.000000