[00:14:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 673761.000000 [00:14:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111617 MB (18% inode=99%): [00:23:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 372139 MB (6% inode=55%): [00:27:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:31:36] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:38:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:43:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:44:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:53:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:59:36] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:06:26] Load avg. on willow is WARNING: WARNING - load average: 15.87, 15.43, 14.56 [01:09:26] Load avg. on willow is OK: OK - load average: 13.82, 14.73, 14.44 [01:14:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 669161.000000 [01:14:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111416 MB (18% inode=99%): [01:16:26] Load avg. on willow is WARNING: WARNING - load average: 15.41, 15.41, 14.93 [01:23:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 374877 MB (7% inode=55%): [01:27:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:31:36] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:36:26] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 44.27, 38.76, 21.50 [01:38:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:40:26] Load avg. on ortelius is WARNING: WARNING - load average: 8.73, 23.88, 19.39 [01:43:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:44:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:47:27] Load avg. on ortelius is OK: OK - load average: 5.68, 10.61, 14.65 [01:53:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:59:36] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:02:26] Load avg. on ortelius is WARNING: WARNING - load average: 12.48, 15.86, 14.94 [02:06:26] Load avg. on willow is WARNING: WARNING - load average: 15.19, 15.64, 15.14 [02:09:26] Load avg. on willow is OK: OK - load average: 14.22, 14.96, 14.95 [02:14:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 666988.000000 [02:14:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109437 MB (18% inode=99%): [02:16:26] Load avg. on willow is WARNING: WARNING - load average: 16.94, 16.18, 15.61 [02:18:16] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 79669 MB (8% inode=99%): [02:23:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 373427 MB (7% inode=55%): [02:27:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:31:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:38:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:44:26] Load avg. on willow is OK: OK - load average: 13.60, 14.42, 14.94 [02:44:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:52:26] Load avg. on ortelius is WARNING: WARNING - load average: 20.21, 17.89, 14.88 [02:53:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:59:37] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:00:26] Load avg. on ortelius is OK: OK - load average: 7.92, 14.38, 14.72 [03:06:26] Load avg. on willow is WARNING: WARNING - load average: 15.73, 15.63, 15.03 [03:14:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 665907.000000 [03:14:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111099 MB (18% inode=99%): [03:23:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 371622 MB (6% inode=55%): [03:27:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:31:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:38:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:43:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:44:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:53:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:59:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:00:25] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.36, 20.07, 17.73 [04:01:38] Load avg. on willow is WARNING: WARNING - load average: 27.52, 21.32, 18.37 [04:14:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 664122.000000 [04:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110754 MB (18% inode=99%): [04:23:39] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 370646 MB (6% inode=55%): [04:27:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:31:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:38:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:44:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:44:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:45:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 53.74, 24.97, 19.95 [04:47:38] Load avg. on willow is WARNING: WARNING - load average: 20.09, 21.90, 19.43 [04:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:14:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 663363.000000 [05:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110499 MB (18% inode=99%): [05:23:39] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 371228 MB (6% inode=55%): [05:27:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:31:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:38:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:44:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:44:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:47:38] Load avg. on willow is WARNING: WARNING - load average: 14.77, 16.50, 16.60 [05:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 662911.000000 [06:14:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109955 MB (18% inode=99%): [06:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 382129 MB (7% inode=55%): [06:27:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:32:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:38:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:44:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:47:38] Load avg. on willow is WARNING: WARNING - load average: 14.78, 15.56, 16.08 [06:54:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 665113.000000 [07:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109199 MB (17% inode=99%): [07:23:39] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 380748 MB (7% inode=55%): [07:27:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:28:39] Load avg. on willow is OK: OK - load average: 13.47, 14.21, 14.93 [07:32:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:36:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 948 MB (6% inode=99%): [07:38:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:38] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1900 MB (11% inode=99%): [07:44:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:44:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:45:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1545 MB (9% inode=99%): [07:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:06:39] Load avg. on willow is WARNING: WARNING - load average: 15.98, 16.09, 15.56 [08:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 667670.000000 [08:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109033 MB (17% inode=99%): [08:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 378112 MB (7% inode=55%): [08:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:27:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:29:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 173 MB (1% inode=99%): [08:32:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:38:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:43:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:45:38] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:45:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:46:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 454 MB (2% inode=99%): [08:54:39] Load avg. on willow is OK: OK - load average: 14.14, 14.60, 14.95 [08:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:06:38] Load avg. on willow is WARNING: WARNING - load average: 15.57, 15.81, 15.43 [09:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 670073.000000 [09:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108884 MB (17% inode=99%): [09:20:08] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 375797 MB (7% inode=55%): [09:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:27:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:27:58] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.027 second response time [09:29:38] Load avg. on willow is OK: OK - load average: 13.95, 14.54, 14.98 [09:30:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:31:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:32:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:36:38] Load avg. on willow is WARNING: WARNING - load average: 15.17, 15.50, 15.44 [09:38:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:38:59] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.737 second response time [09:42:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:45:48] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:46:48] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 136 MB (0% inode=99%): [09:48:48] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:04:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:09:38] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2567 MB (13% inode=99%): [10:13:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1683 MB (9% inode=99%): [10:14:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 672522.000000 [10:14:39] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2521 MB (13% inode=99%): [10:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108684 MB (17% inode=99%): [10:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 374944 MB (7% inode=55%): [10:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:27:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:32:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:36:39] Load avg. on willow is WARNING: WARNING - load average: 16.02, 16.02, 15.94 [10:38:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:44:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:53:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1969 MB (10% inode=99%): [10:54:38] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2518 MB (13% inode=99%): [10:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:04:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 674399.000000 [11:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108530 MB (17% inode=99%): [11:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 374280 MB (7% inode=55%): [11:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:27:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1395 MB (7% inode=99%): [11:28:38] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2203 MB (11% inode=99%): [11:28:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:30:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1995 MB (10% inode=99%): [11:32:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:34:39] Load avg. on willow is OK: OK - load average: 13.75, 14.50, 14.98 [11:38:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:46:38] Load avg. on willow is WARNING: WARNING - load average: 14.98, 14.99, 15.04 [11:47:38] Load avg. on willow is OK: OK - load average: 14.55, 14.84, 14.98 [11:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:04:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 674416.000000 [12:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108304 MB (17% inode=99%): [12:16:38] Load avg. on willow is WARNING: WARNING - load average: 17.89, 18.00, 16.81 [12:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 372584 MB (6% inode=55%): [12:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:28:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:32:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:38:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 2036 MB (10% inode=99%): [12:38:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:39:38] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2161 MB (11% inode=99%): [12:40:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1763 MB (9% inode=99%): [12:44:38] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2076 MB (11% inode=99%): [12:44:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:04:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 676455.000000 [13:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108067 MB (17% inode=99%): [13:16:38] Load avg. on willow is WARNING: WARNING - load average: 14.57, 14.80, 15.09 [13:17:38] Load avg. on willow is OK: OK - load average: 13.23, 14.36, 14.91 [13:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 369870 MB (6% inode=55%): [13:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:27:39] Free Memory on damiana is WARNING: WARNING - 5.4% (449556 kB) free! [13:28:38] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (395632 kB) free! [13:28:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:29:38] Free Memory on damiana is WARNING: WARNING - 6.2% (522464 kB) free! [13:31:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1983 MB (10% inode=99%): [13:32:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:38:39] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2070 MB (11% inode=99%): [13:38:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:39:08] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:39:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1646 MB (9% inode=99%): [13:39:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:44:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:54:38] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2106 MB (11% inode=99%): [13:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:04:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 676437.000000 [14:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107881 MB (17% inode=99%): [14:18:18] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 80002 MB (8% inode=99%): [14:20:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 934 MB (5% inode=99%): [14:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 367523 MB (6% inode=55%): [14:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:32:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:38:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:40:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:04:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 678040.000000 [15:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107624 MB (17% inode=99%): [15:20:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 890 MB (4% inode=99%): [15:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 364860 MB (6% inode=55%): [15:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:32:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:39:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:41:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:44:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:59:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:04:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:07:57] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.356 second response time [16:08:58] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.681 second response time [16:09:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:09:58] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.347 second response time [16:10:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 679990.000000 [16:14:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107400 MB (17% inode=99%): [16:20:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 180 MB (0% inode=99%): [16:23:39] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 361799 MB (6% inode=55%): [16:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:32:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:39:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:41:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:43:38] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:45:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 147 MB (0% inode=99%): [16:46:39] Load avg. on willow is WARNING: WARNING - load average: 15.74, 15.30, 14.95 [16:46:39] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:47:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:48:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:49:59] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:52:38] Load avg. on willow is OK: OK - load average: 14.39, 14.95, 14.90 [16:59:38] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3043 MB (10% inode=93%): [16:59:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:59:59] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3067 MB (10% inode=93%): [17:04:58] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.794 second response time [17:12:59] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.136 second response time [17:14:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 681489.000000 [17:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107266 MB (17% inode=99%): [17:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359793 MB (6% inode=55%): [17:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:25:58] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.879 second response time [17:26:58] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.433 second response time [17:27:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:28:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 164 MB (0% inode=99%): [17:32:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:39:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:41:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:42:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:42:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:43:38] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:43:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:44:38] Load avg. on willow is WARNING: WARNING - load average: 14.94, 15.21, 15.02 [17:44:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 240 MB (1% inode=99%): [17:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:46:38] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:47:38] Load avg. on willow is OK: OK - load average: 14.48, 14.95, 14.95 [17:53:08] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:38] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3023 MB (10% inode=92%): [17:59:59] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3029 MB (10% inode=92%): [18:00:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:01:38] Load avg. on willow is WARNING: WARNING - load average: 26.63, 21.04, 17.57 [18:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 683233.000000 [18:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107072 MB (17% inode=99%): [18:16:58] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.805 second response time [18:17:59] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.005 second response time [18:23:39] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 357869 MB (6% inode=55%): [18:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:24:58] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.494 second response time [18:30:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 186 MB (1% inode=99%): [18:31:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:33:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:33:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:34:48] Sun Grid Engine execd on ortelius is CRITICAL: Connection refused by host [18:39:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:40:38] / on ortelius is CRITICAL: Connection refused by host [18:40:39] DiskSuite on ortelius is CRITICAL: Connection refused by host [18:40:39] Load avg. on ortelius is CRITICAL: Connection refused by host [18:40:54] Environment IPMI on ortelius is CRITICAL: Connection refused by host [18:42:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:43:37] hey folks, everything looks kosher on http://status.toolserver.org but the TS web server is extremely slow and multiple people are getting Bad Gateway errors [18:44:15] same here [18:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:45:43] duh: I just mailed the ts-admins [18:45:56] ah thanks [18:46:15] it was only a one off thing for me, so i just assumed it fixed itself [18:47:57] duh: still not working on my end [18:49:17] hm i just got another one [18:50:38] Load avg. on ortelius is OK: OK - load average: 1.77, 1.66, 2.11 [18:50:48] Environment IPMI on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:50:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:51:39] / on ortelius is OK: DISK OK - free space: / 13237 MB (44% inode=92%): [18:51:39] DiskSuite on ortelius is OK: OK - No disk failures detected [18:51:58] Environment IPMI on ortelius is OK: ok: temperature ok fan ok voltage ok chassis ok [18:52:58] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:54:39] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:56:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 225 MB (1% inode=99%): [18:59:38] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3011 MB (10% inode=92%): [18:59:58] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3015 MB (10% inode=92%): [19:01:39] Load avg. on willow is WARNING: WARNING - load average: 16.93, 15.79, 15.80 [19:01:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:01:59] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.066 second response time [19:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 685352.000000 [19:14:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106879 MB (17% inode=99%): [19:19:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:21:39] /tmp on ortelius is OK: DISK OK - free space: /tmp 18951 MB (95% inode=99%): [19:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356012 MB (6% inode=55%): [19:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:33:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:39:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:43:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:56:39] Load avg. on willow is OK: OK - load average: 14.28, 14.61, 14.98 [19:59:38] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2913 MB (9% inode=93%): [19:59:58] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2921 MB (9% inode=93%): [20:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:10:38] Load avg. on willow is WARNING: WARNING - load average: 16.25, 15.29, 15.12 [20:14:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 687124.000000 [20:14:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106660 MB (17% inode=99%): [20:19:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:23:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352799 MB (6% inode=55%): [20:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:29:39] Load avg. on willow is OK: OK - load average: 13.39, 14.56, 14.96 [20:33:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [20:36:39] Load avg. on willow is WARNING: WARNING - load average: 14.96, 14.94, 15.03 [20:37:38] Load avg. on willow is OK: OK - load average: 13.81, 14.61, 14.91 [20:39:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:43:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:44:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:52:18] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 70.42, 41.22, 21.17 [21:00:08] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 714 MB (2% inode=93%): [21:00:09] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 710 MB (2% inode=93%): [21:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:11:58] Load avg. on willow is WARNING: WARNING - load average: 17.17, 15.90, 15.30 [21:15:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 689144.000000 [21:15:38] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106372 MB (17% inode=99%): [21:20:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:21:58] Load avg. on willow is OK: OK - load average: 14.21, 14.76, 14.98 [21:22:19] Load avg. on ortelius is WARNING: WARNING - load average: 10.83, 13.54, 19.97 [21:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:24:19] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349154 MB (6% inode=55%): [21:29:18] Load avg. on ortelius is OK: OK - load average: 3.02, 7.32, 14.78 [21:34:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:40:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:44:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:45:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:55:52] nighty [22:00:48] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 417 MB (1% inode=93%): [22:00:58] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 416 MB (1% inode=93%): [22:02:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:14:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 691326.000000 [22:15:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106165 MB (17% inode=99%): [22:20:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:24:18] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348146 MB (6% inode=55%): [22:34:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:40:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:45:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:00:48] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 395 MB (1% inode=93%): [23:00:58] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 396 MB (1% inode=93%): [23:02:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:14:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 692951.000000 [23:15:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105883 MB (17% inode=99%): [23:20:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:24:18] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 346160 MB (6% inode=55%): [23:34:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:40:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:44:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:45:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default