[00:00:33] @replag [00:00:33] Brooke: s1-rr-a: 5h 8m 48s [-1.83 s/s]; s1-user: 8h 9m 0s [-0.11 s/s]; s3-rr-a: 1m 22s [-0.35 s/s]; s3-user: 1m 22s [-0.35 s/s] [00:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:16:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105986 MB (17% inode=99%): [00:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:18:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16572 [00:23:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:25:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15929.000000 [00:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:31:28] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [00:32:28] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [00:33:26] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [00:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:37:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:37:07] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:37:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:38:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:41:06] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1409 MB (8% inode=99%): [00:42:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30274 [00:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:45:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30353.000000 [00:58:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201004 MB (20% inode=99%): [01:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105828 MB (17% inode=99%): [01:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:18:28] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9262 [01:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:25:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8611.000000 [01:26:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [01:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [01:33:29] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [01:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:37:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:37:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:38:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:41:08] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 442 MB (2% inode=99%): [01:42:15] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32003 [01:43:18] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:43:26] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:44:06] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1614 MB (9% inode=99%): [01:44:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:44:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:45:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32045.000000 [01:52:06] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1970 MB (11% inode=99%): [02:00:06] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1761 MB (10% inode=99%): [02:05:59] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:06:56] RAID on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:08:35] RAID on rosemary is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:09:18] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3568.000000 [02:09:26] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3551 [02:16:17] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105744 MB (17% inode=99%): [02:16:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:26:16] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1663.000000 [02:26:26] MySQL slave on thyme is OK: Uptime: 19298 Threads: 7 Questions: 7260769 Slow queries: 648 Opens: 283 Flush tables: 1 Open tables: 152 Queries per second avg: 376.244 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1632 [02:28:06] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1738 MB (10% inode=99%): [02:31:26] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [02:32:26] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [02:33:26] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [02:36:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:36:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:37:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:40:06] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1929 MB (11% inode=99%): [02:42:18] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33583 [02:44:11] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1650 MB (10% inode=99%): [02:44:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:44:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:46:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33684.000000 [02:57:10] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1933 MB (11% inode=99%): [03:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:16:28] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105673 MB (17% inode=99%): [03:16:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:23:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:31:31] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [03:32:31] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [03:33:31] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [03:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:37:00] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:37:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:38:21] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:42:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34019 [03:44:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:46:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34109.000000 [03:53:40] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:53:41] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:10] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 65114 MB (6% inode=97%): [03:54:10] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 65114 MB (6% inode=97%): [04:06:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105550 MB (17% inode=99%): [04:17:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:23:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:31:42] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [04:32:40] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [04:33:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [04:36:13] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:37:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:39:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:42:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34653 [04:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:46:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34679.000000 [05:06:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105416 MB (17% inode=99%): [05:17:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:23:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:31:41] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [05:32:40] APT on mayapple is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [05:33:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [05:36:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:37:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:42:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35274 [05:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:46:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35404.000000 [06:06:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105285 MB (17% inode=99%): [06:17:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:24:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:31:42] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [06:33:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [06:33:41] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [06:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:37:22] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:42:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36108 [06:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:46:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36128.000000 [06:48:11] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1669 MB (10% inode=99%): [06:54:10] /tmp on ortelius is OK: DISK OK - free space: /tmp 12456 MB (75% inode=99%): [07:06:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105148 MB (17% inode=99%): [07:17:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:24:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:31:42] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [07:33:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [07:33:41] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [07:36:13] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:37:22] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:38:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:41:11] /sql on thyme is OK: DISK OK - free space: /sql 201812 MB (21% inode=99%): [07:42:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36441 [07:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:46:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36488.000000 [07:48:11] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201337 MB (20% inode=99%): [08:06:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104991 MB (17% inode=99%): [08:17:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:24:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:31:41] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [08:33:41] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [08:34:40] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [08:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:37:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:42:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35654 [08:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:46:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35741.000000 [09:06:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:17:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104744 MB (17% inode=99%): [09:17:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:19:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 94856 MB (9% inode=99%): [09:24:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:30:01] Environment IPMI on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:30:33] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:31:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:31:42] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [09:32:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:33:41] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [09:35:31] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [09:36:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:37:03] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:42:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35352 [09:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:46:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35539.000000 [10:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104495 MB (17% inode=99%): [10:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:31:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:31:44] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [10:32:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:40] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [10:35:40] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [10:36:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:37:05] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:37:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:42:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35245 [10:43:41] anybody here who would be able to add jars to Tomcats lib? [10:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:46:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35366.000000 [11:09:41] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 31963 MB (7% inode=99%): [11:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104281 MB (17% inode=99%): [11:25:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:31:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:31:43] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [11:32:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:33:40] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [11:35:40] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [11:36:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:37:20] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:38:34] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:39:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:42:09] ant * [Toolserver-l] Database connection problem [11:42:20] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36586 [11:44:30] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:44:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:47:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36784.000000 [11:48:09] Platonides * Re: [Toolserver-l] Database connection problem [11:50:09] Marlen Caemmerer * Re: [Toolserver-l] Database connection problem [12:02:33] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:05:09] ant * Re: [Toolserver-l] Database connection problem [12:17:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104078 MB (17% inode=99%): [12:25:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [12:32:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:33:40] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [12:35:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [12:36:12] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:37:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:37:10] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:37:22] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:42:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36665 [12:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:47:21] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36725.000000 [13:03:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103844 MB (17% inode=99%): [13:25:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [13:32:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:33:40] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [13:35:40] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [13:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:37:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:37:15] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:37:20] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:37:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:39:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:42:20] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38825 [13:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:48:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38968.000000 [14:04:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103569 MB (17% inode=99%): [14:26:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:31:07] Why people don't point to the wiki when answering the mailing list is beyond me. https://wiki.toolserver.org/view/Database_access#C [14:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [14:32:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:33:40] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [14:35:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [14:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:37:00] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:37:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:37:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:38:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:39:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:42:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39573 [14:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:48:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39602.000000 [15:04:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:17:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103326 MB (17% inode=99%): [15:26:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [15:32:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:33:43] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [15:35:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [15:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:37:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:37:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:38:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:42:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40645 [15:44:32] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:44:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:48:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40914.000000 [15:54:10] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 67049 MB (6% inode=98%): [15:54:11] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 67049 MB (6% inode=98%): [16:04:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103047 MB (16% inode=99%): [16:26:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:31:44] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [16:32:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:33:40] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [16:35:42] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [16:36:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:37:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:42:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41883 [16:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:48:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41918.000000 [17:04:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102730 MB (16% inode=99%): [17:27:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:28:01] RAID on rosemary is CRITICAL: Connection refused by host [17:29:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [17:31:45] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [17:32:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:33:41] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [17:34:04] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [17:34:04] Environment IPMI on rosemary is CRITICAL: Connection refused by host [17:34:10] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [17:34:11] SMTP on rosemary is CRITICAL: Connection refused [17:34:11] SSH on rosemary is CRITICAL: Connection refused [17:34:21] / on rosemary is CRITICAL: Connection refused by host [17:34:21] /tmp on rosemary is CRITICAL: Connection refused by host [17:34:32] Load avg. on rosemary is CRITICAL: Connection refused by host [17:34:41] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [17:35:43] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [17:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:42:21] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [17:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:48:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [18:05:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102461 MB (16% inode=99%): [18:27:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:31:41] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [18:32:32] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:32:32] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/10:DOWN: 1 int NOK : CRITICAL [18:33:21] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:DOWN: 1 int NOK : CRITICAL [18:33:43] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [18:35:40] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [18:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:36:48] hello [18:36:55] enwiki-user died [18:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:37:08] i am troubleshooting [18:37:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:37:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:37:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:44:30] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:44:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:52:29] hi nosy. sounds like again it's a bad point in time to point to "my" jira request about missing java libraries... [18:55:20] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/6:UP:1 UP: OK [18:55:31] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/10:UP:1 UP: OK [19:00:44] ok...s1 should be back soon [19:00:59] mysql is starting...takes some time [19:01:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [19:02:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [19:02:01] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [19:02:12] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [19:02:54] hello all [19:02:58] hi DaBPunkt [19:05:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:05:49] hello DaBPunkt [19:05:57] rosemary is just coming back [19:07:00] Could not connect to enwiki-p.userdb.toolserver.org : Lost connection to MySQL server at 'reading initial communication packet', system error: 0 is something broken? [19:13:34] nosy: Any eta on when noresmary will be back? just curious [19:14:09] legoktm might be 5 or 10 minutes [19:14:09] ok thanks [19:14:10] innodb is still working through its logs [19:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102179 MB (16% inode=99%): [19:24:01] MySQL on rosemary is OK: Uptime: 1535 Threads: 47 Questions: 519 Slow queries: 2 Opens: 25 Flush tables: 1 Open tables: 18 Queries per second avg: 0.338 [19:27:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [19:32:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:33:40] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [19:35:40] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [19:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:37:00] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:37:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:37:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:38:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:39:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:40:28] what can I do to get anyone to copy missing jars to the tomcat lib (https://jira.toolserver.org/browse/TS-1550)? [19:40:30] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:41:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:44:30] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:48:11] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200689 MB (20% inode=99%): [19:51:10] /sql on thyme is OK: DISK OK - free space: /sql 201734 MB (21% inode=99%): [19:58:15] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200752 MB (20% inode=99%): [20:02:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50612.000000 [20:02:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50605 [20:02:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9606.000000 [20:17:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101873 MB (16% inode=99%): [20:28:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:31:40] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [20:32:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:33:40] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [20:35:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [20:36:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:00] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:41:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:44:34] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:44:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:02:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47594.000000 [21:02:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47602 [21:02:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13206.000000 [21:17:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101575 MB (16% inode=99%): [21:19:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 94505 MB (9% inode=99%): [21:28:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:31:41] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [21:32:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:33:41] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [21:35:44] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [21:36:13] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:37:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:38:34] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:39:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:42:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:53:17] nacht ts [22:01:31] Load avg. on ortelius is WARNING: WARNING - load average: 7.98, 15.23, 11.45 [22:02:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43264.000000 [22:02:11] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43247 [22:02:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16806.000000 [22:02:34] Load avg. on ortelius is OK: OK - load average: 5.23, 13.17, 10.96 [22:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101321 MB (16% inode=99%): [22:28:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:31:44] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [22:32:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:33:41] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [22:35:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [22:36:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:37:02] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:39:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:42:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:02:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40265.000000 [23:02:14] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40255 [23:02:14] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20406.000000 [23:09:44] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 31730 MB (7% inode=99%): [23:17:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101012 MB (16% inode=99%): [23:28:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:31:41] APT on nightshade is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [23:32:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:33:42] APT on mayapple is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [23:35:41] APT on yarrow is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [23:36:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:37:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:37:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:37:11] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:37:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:37:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:39:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:43:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:44:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:44:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output