[03:51:20] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:55:41] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107036 MB (11% inode=99%): [03:56:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106432 MB (10% inode=99%): [03:58:14] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (could not connect to server: Connection refused [03:58:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70014 MB (11% inode=99%): [04:06:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:06:27] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:06:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:14:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:20:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:21:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:23:03] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:25:01] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:26:18] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:27:01] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:27:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:29:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:39:08] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:54:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:58:19] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (could not connect to server: Connection refused [04:58:38] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70014 MB (11% inode=99%): [05:06:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:07:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:14:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:20:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:21:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:23:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:25:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:26:40] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:27:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:27:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:29:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:39:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:47:29] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1077 MB (7% inode=99%): [05:54:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:58:19] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (could not connect to server: Connection refused [05:58:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70014 MB (11% inode=99%): [06:07:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:08:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:14:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:20:20] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:21:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:23:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:26:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:26:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:27:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:27:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:29:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:31:21] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:31:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:36:08] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:36:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:39:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:39:31] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:40:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:43:11] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:43:20] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:44:28] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:31] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 806 MB (5% inode=99%): [06:58:19] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (could not connect to server: Connection refused [06:58:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70014 MB (11% inode=99%): [07:07:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:15:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:17:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:20:20] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [07:21:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:24:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:25:00] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:26:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:26:42] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:27:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:27:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:30:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:39:23] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:39:28] /tmp on ortelius is OK: DISK OK - free space: /tmp 9851 MB (69% inode=99%): [07:51:39] /sql on ptolemy is CRITICAL: Connection refused by host [08:01:40] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199043 MB (20% inode=99%): [08:02:23] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (could not connect to server: Connection refused [08:02:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70014 MB (11% inode=99%): [08:07:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:09:18] PostgreSQL on ptolemy is OK: OK - database osm_mapnik (0 sec.) [08:15:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:16:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:21:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:21:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [08:24:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:24:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:26:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:26:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:27:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:27:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:30:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:39:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:02:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69624 MB (11% inode=99%): [09:07:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:17:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:21:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:21:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [09:24:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:25:01] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:26:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:26:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:27:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:27:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:40:28] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:02:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69384 MB (11% inode=99%): [10:07:51] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:16:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:17:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:21:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:22:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [10:24:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:25:00] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:26:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:26:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:27:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:27:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:31:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:41:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:02:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69063 MB (11% inode=99%): [11:07:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:16:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:17:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:21:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:22:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [11:24:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:24:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:26:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:26:41] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:27:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:27:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:31:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:32:18] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:32:31] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:32:38] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:33:31] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 62515 MB (6% inode=97%): [11:33:39] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 62515 MB (6% inode=97%): [11:34:19] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 30102 MB (7% inode=99%): [11:35:19] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:35:32] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:35:38] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:41:18] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:48:07] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [11:48:21] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [11:59:20] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 30076 MB (7% inode=99%): [12:02:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68727 MB (11% inode=99%): [12:07:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:07] MySQL on z-dat-s3-a is OK: Uptime: 1622 Threads: 15 Questions: 203 Slow queries: 0 Opens: 89 Flush tables: 1 Open tables: 70 Queries per second avg: 0.125 [12:09:19] MySQL slave on z-dat-s3-a is OK: Uptime: 1627 Threads: 19 Questions: 270 Slow queries: 0 Opens: 123 Flush tables: 1 Open tables: 104 Queries per second avg: 0.165 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1639 [12:11:30] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 63566 MB (6% inode=97%): [12:11:39] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 63566 MB (6% inode=97%): [12:16:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:18:18] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2084 [12:19:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:21:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:22:36] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [12:25:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:25:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:26:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:26:45] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:27:42] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:27:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:31:24] MySQL slave on z-dat-s3-a is OK: Uptime: 2947 Threads: 16 Questions: 777138 Slow queries: 128 Opens: 17710 Flush tables: 1 Open tables: 9726 Queries per second avg: 263.704 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1771 [12:32:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:42:24] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:02:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68432 MB (11% inode=99%): [13:07:50] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:16:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:18:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:21:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:22:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:25:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:26:00] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:26:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:27:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:27:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:27:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:32:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:42:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:03:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68042 MB (11% inode=99%): [14:07:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:16:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:19:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:21:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:22:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [14:25:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:26:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:26:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:27:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:27:42] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:28:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:32:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:42:18] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:03:50] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67702 MB (11% inode=99%): [15:08:01] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:21:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:21:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:22:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [15:26:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:26:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:26:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:27:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:27:40] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:28:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:32:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:36:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107108 MB (11% inode=99%): [15:37:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106521 MB (10% inode=99%): [15:42:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:03:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67302 MB (11% inode=99%): [16:07:59] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:17:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:21:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:21:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:22:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [16:26:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:26:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:26:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:27:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:27:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:28:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:32:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:35:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67085 MB (10% inode=99%): [16:42:18] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:07:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:17:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:22:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:22:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:22:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [17:26:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:26:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:27:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:27:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:28:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:28:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:33:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:35:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66769 MB (10% inode=99%): [17:42:23] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:09:00] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:17:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:22:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:22:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [18:23:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:26:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:26:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:27:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:27:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:28:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:28:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:33:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:35:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66466 MB (10% inode=99%): [18:42:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:08:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:22:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:22:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [19:23:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:26:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:26:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:27:40] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:27:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:28:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:28:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:33:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:35:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66162 MB (10% inode=99%): [19:42:18] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:01:49] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198508 MB (20% inode=99%): [20:08:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:22:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [20:23:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:23:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:26:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:27:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:27:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:28:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:29:02] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:29:02] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:33:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:36:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65839 MB (10% inode=99%): [20:42:28] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:08:59] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:19:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:22:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [21:24:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:24:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:26:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:27:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:28:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:28:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:30:00] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:30:00] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:33:47] Hm.. prompt color is gone on willow? [21:33:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:34:08] Danny_B|backup: Do you have a PS1 with colors? [21:34:15] Mine doesn't work on willow, but does work on nightsade [21:34:23] Nightshade* [21:34:26] Tried logging in again but same result [21:35:40] Merlissimo: Any idea? [21:35:51] especially annoying since login.toolserver.org defaults to willow [21:37:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65489 MB (10% inode=99%): [21:43:02] Krinkle: i do not have a colored promt, but grep --color worked for me as a quick test [21:43:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:01:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107115 MB (11% inode=99%): [22:03:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106643 MB (10% inode=99%): [22:08:59] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:19:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:23:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [22:24:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:24:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:26:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:27:42] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:28:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:28:11] btw, solved the issue with my terminal colors [22:28:17] /usr/bin/tput didn't detect my terminal name [22:28:38] MacOSX update changed the default to "xterm-256colors" which /usr/bin/tput didn't recognize [22:28:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:28:53] changed it in Mac Terminal preferences to name itself rxvt [22:29:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:29:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:33:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:37:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65110 MB (10% inode=99%): [22:43:18] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:45:50] Krinkle: i do [22:46:05] (as long as ps1 == what i see in putty) [22:46:22] indeed [22:46:31] It was a Mac issue [22:46:46] combined with /usr/bin/tput on Solaris not recognizing xterm-256colors [22:47:03] Ubuntu recognises it fine [23:09:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:19:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:20:32] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1349 MB (11% inode=99%): [23:21:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:23:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [23:24:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:24:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:26:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:27:49] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:28:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:28:28] /tmp on ortelius is OK: DISK OK - free space: /tmp 2846 MB (21% inode=99%): [23:29:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:30:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:30:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:33:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:35:28] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2696 MB (20% inode=99%): [23:37:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64876 MB (10% inode=99%): [23:43:18] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:54:32] /tmp on ortelius is OK: DISK OK - free space: /tmp 2847 MB (21% inode=99%): [23:59:19] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 30176 MB (7% inode=99%):