[00:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340766 MB (6% inode=33%): [00:19:52] I just hit 100% CPU on Toolserver! [00:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 108071 MB (17% inode=99%): [00:19:59] * wolfgang42 is strangely excited [00:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:40:55] whym_away: depending on the server 100% means only 1 CPU and you can in theory use more than 100% [00:48:22] nacht ts [00:49:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.12, 25.18, 26.13 [00:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:54:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:54:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:56:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [01:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 342049 MB (6% inode=33%): [01:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107882 MB (17% inode=99%): [01:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:29:12] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1954 [01:29:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1956.000000 [01:49:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.71, 24.04, 24.20 [01:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:54:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:54:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:56:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:57:42] /tmp on willow is WARNING: DISK WARNING - free space: / 21955 MB (20% inode=98%): [01:57:42] / on willow is WARNING: DISK WARNING - free space: / 21955 MB (20% inode=98%): [02:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340508 MB (6% inode=33%): [02:19:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106016 MB (17% inode=99%): [02:23:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:29:12] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2135 [02:29:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2132.000000 [02:49:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:50:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 89760 MB (9% inode=99%): [02:53:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.46, 24.52, 25.27 [02:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:54:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:54:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:56:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:57:42] /tmp on willow is WARNING: DISK WARNING - free space: / 21881 MB (20% inode=98%): [02:57:42] / on willow is WARNING: DISK WARNING - free space: / 21881 MB (20% inode=98%): [03:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:08:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 339045 MB (6% inode=33%): [03:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107718 MB (17% inode=99%): [03:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:29:12] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1928 [03:29:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1926.000000 [03:49:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.47, 29.65, 29.96 [03:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:54:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:54:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:56:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:57:43] /tmp on willow is WARNING: DISK WARNING - free space: / 21778 MB (20% inode=98%): [03:57:43] / on willow is WARNING: DISK WARNING - free space: / 21778 MB (20% inode=98%): [04:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:08:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 337693 MB (6% inode=33%): [04:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107657 MB (17% inode=99%): [04:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:29:12] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2456 [04:29:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2456.000000 [04:49:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.86, 29.52, 30.43 [04:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:54:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:54:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:56:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [04:57:43] / on willow is WARNING: DISK WARNING - free space: / 21730 MB (20% inode=98%): [04:57:43] /tmp on willow is WARNING: DISK WARNING - free space: / 21730 MB (20% inode=98%): [05:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 335917 MB (6% inode=33%): [05:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107565 MB (17% inode=99%): [05:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:29:12] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2569 [05:29:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2568.000000 [05:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.94, 34.50, 33.30 [05:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:56:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [05:57:42] / on willow is WARNING: DISK WARNING - free space: / 21675 MB (20% inode=98%): [05:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21675 MB (20% inode=98%): [06:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338143 MB (6% inode=33%): [06:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107474 MB (17% inode=99%): [06:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:28:12] MySQL slave on rosemary is OK: Uptime: 2105225 Threads: 17 Questions: 1424034877 Slow queries: 434001 Opens: 132694 Flush tables: 1 Open tables: 3716 Queries per second avg: 676.428 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1792 [06:28:22] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1791.000000 [06:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.66, 29.88, 29.81 [06:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:56:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:57:42] / on willow is WARNING: DISK WARNING - free space: / 21610 MB (20% inode=98%): [06:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21610 MB (20% inode=98%): [07:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 337456 MB (6% inode=33%): [07:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107358 MB (17% inode=99%): [07:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.76, 26.79, 26.46 [07:54:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:57:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [07:57:42] / on willow is WARNING: DISK WARNING - free space: / 21545 MB (20% inode=98%): [07:57:51] /tmp on willow is WARNING: DISK WARNING - free space: / 21545 MB (20% inode=98%): [08:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 335060 MB (6% inode=32%): [08:19:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107242 MB (17% inode=99%): [08:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.09, 26.81, 27.44 [08:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:57:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:57:42] / on willow is WARNING: DISK WARNING - free space: / 21477 MB (20% inode=98%): [08:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21477 MB (20% inode=98%): [09:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332969 MB (6% inode=32%): [09:20:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107073 MB (17% inode=99%): [09:23:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.30, 26.54, 27.09 [09:54:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:57:31] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:57:43] / on willow is WARNING: DISK WARNING - free space: / 21417 MB (20% inode=98%): [09:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21417 MB (20% inode=98%): [10:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331670 MB (6% inode=32%): [10:20:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106920 MB (17% inode=99%): [10:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:27:42] /sql on cassia is WARNING: DISK WARNING - free space: /sql 93891 MB (7% inode=99%): [10:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.97, 24.79, 24.66 [10:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:54:42] Load avg. on ptolemy is WARNING: WARNING - load average: 24.78, 19.90, 14.70 [10:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:57:42] / on willow is WARNING: DISK WARNING - free space: / 21347 MB (20% inode=98%): [10:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21347 MB (20% inode=98%): [10:58:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [11:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:04:42] Load avg. on ptolemy is OK: OK - load average: 7.39, 13.74, 14.75 [11:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 334194 MB (6% inode=32%): [11:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106719 MB (17% inode=99%): [11:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.69, 23.84, 23.77 [11:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:57:42] / on willow is WARNING: DISK WARNING - free space: / 21281 MB (20% inode=98%): [11:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21281 MB (20% inode=98%): [11:58:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [12:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331531 MB (6% inode=32%): [12:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106512 MB (17% inode=99%): [12:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.98, 25.05, 25.77 [12:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:57:42] / on willow is WARNING: DISK WARNING - free space: / 21220 MB (20% inode=98%): [12:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 21220 MB (20% inode=98%): [12:58:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [13:03:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 328385 MB (6% inode=32%): [13:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106299 MB (17% inode=99%): [13:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.61, 23.54, 23.24 [13:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:57:42] / on willow is WARNING: DISK WARNING - free space: / 21167 MB (20% inode=98%): [13:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21167 MB (20% inode=98%): [13:59:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [14:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 329464 MB (6% inode=32%): [14:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106012 MB (17% inode=99%): [14:23:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:50:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 89272 MB (9% inode=99%): [14:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.57, 23.84, 24.16 [14:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:57:42] / on willow is WARNING: DISK WARNING - free space: / 21100 MB (20% inode=98%): [14:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21100 MB (20% inode=98%): [15:00:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [15:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 328223 MB (6% inode=32%): [15:20:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105799 MB (17% inode=99%): [15:22:21] hello all [15:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.09, 24.25, 24.25 [15:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:54:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:57:42] / on willow is WARNING: DISK WARNING - free space: / 21044 MB (20% inode=98%): [15:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 21043 MB (20% inode=98%): [16:00:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [16:03:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:08:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 326946 MB (6% inode=32%): [16:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105555 MB (17% inode=99%): [16:22:04] DaB. * [Toolserver-announce] Downtime of s7 starting tomorrow [16:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:37:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2032.000000 [16:49:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.50, 23.45, 24.07 [16:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:55:53] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1739.000000 [16:57:14] @replag [16:57:14] DaBPunkt: s2-user-c: 27m 38s [+0.02 s/s]; s2-user-wd: 10s [-0.00 s/s]; s3-user: 29s [+0.00 s/s]; s5-user-c: 27m 38s [+0.01 s/s]; s5-user-wd: 10s [-0.00 s/s] [16:57:42] / on willow is WARNING: DISK WARNING - free space: / 20987 MB (20% inode=98%): [16:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20986 MB (20% inode=98%): [17:00:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [17:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:08:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325870 MB (6% inode=32%): [17:20:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105332 MB (17% inode=99%): [17:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.88, 26.66, 27.07 [17:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:54:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:57:42] / on willow is WARNING: DISK WARNING - free space: / 20924 MB (19% inode=98%): [17:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20924 MB (19% inode=98%): [18:00:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [18:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 323926 MB (6% inode=32%): [18:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105134 MB (17% inode=99%): [18:24:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:37:42] I am confused about toolserver.org/~dpl/... [18:37:52] / on ptolemy is WARNING: DISK WARNING - free space: / 4170 MB (20% inode=91%): [18:38:58] i cant find the user anywhere [18:41:10] addshore: if with 'the user' you mean 'the corresponding public_html', try /home/project/d/p/l/dpl/public_html [18:41:30] DaBPunkt: Thanks for the email with the cleanup log. It was very useful. [18:41:51] It showed me that what I told you was wrong. Or at least it didn't behave as I expected it to. [18:41:51] cheers valhallasw [18:42:48] I noticed that the expiration process, counter to what I thought currently also modifies atime, not only mtime. Which is something I need to fix. [18:44:14] I'll try and recompile mod_tile first though to fix those notice log messages. Which I'll get to tonight (US time) [18:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.18, 26.96, 27.22 [18:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:57:43] / on willow is WARNING: DISK WARNING - free space: / 20873 MB (19% inode=98%): [18:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 20873 MB (19% inode=98%): [19:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [19:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325901 MB (6% inode=32%): [19:20:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104877 MB (17% inode=99%): [19:21:30] addshore: that's a strange one. Let me see what I can find [19:23:17] addshore: it's a multi-maintainer-project. Its members are jason, russell and dispenser [19:37:52] / on ptolemy is WARNING: DISK WARNING - free space: / 3989 MB (19% inode=91%): [19:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.08, 26.08, 25.88 [19:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:54:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:57:42] / on willow is WARNING: DISK WARNING - free space: / 20816 MB (19% inode=98%): [19:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20816 MB (19% inode=98%): [20:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:03:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324813 MB (6% inode=32%): [20:18:22] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104700 MB (17% inode=99%): [20:28:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:29:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:30:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:52] / on ptolemy is WARNING: DISK WARNING - free space: / 3820 MB (19% inode=91%): [20:49:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.10, 23.98, 24.12 [20:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:57:42] / on willow is WARNING: DISK WARNING - free space: / 20732 MB (19% inode=98%): [20:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20732 MB (19% inode=98%): [21:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:01:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:08:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 323793 MB (6% inode=32%): [21:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104554 MB (17% inode=99%): [21:37:52] / on ptolemy is WARNING: DISK WARNING - free space: / 3660 MB (18% inode=91%): [21:49:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.61, 23.32, 23.16 [21:54:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:57:42] / on willow is WARNING: DISK WARNING - free space: / 20658 MB (19% inode=98%): [21:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20658 MB (19% inode=98%): [22:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:08:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322538 MB (6% inode=32%): [22:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:20:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104229 MB (17% inode=99%): [22:27:42] /sql on cassia is WARNING: DISK WARNING - free space: /sql 96047 MB (8% inode=99%): [22:37:52] / on ptolemy is WARNING: DISK WARNING - free space: / 3546 MB (17% inode=91%): [22:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.10, 21.69, 22.36 [22:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:57:42] / on willow is WARNING: DISK WARNING - free space: / 20581 MB (19% inode=98%): [22:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20581 MB (19% inode=98%): [23:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [23:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321849 MB (6% inode=32%): [23:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104096 MB (17% inode=99%): [23:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:37:52] / on ptolemy is WARNING: DISK WARNING - free space: / 3462 MB (17% inode=91%): [23:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.10, 22.89, 22.62 [23:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:57:42] / on willow is WARNING: DISK WARNING - free space: / 20499 MB (19% inode=98%): [23:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20499 MB (19% inode=98%):