[00:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90968 MB (14% inode=99%): [00:24:07] DeltaQuad Wikipedia * Re: [Toolserver-l] Cron on willow [00:24:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 283815 MB (5% inode=29%): [00:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:51:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [01:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:21:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90801 MB (14% inode=99%): [01:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 288629 MB (5% inode=29%): [01:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:54:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:05:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:21:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88906 MB (14% inode=99%): [02:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 285157 MB (5% inode=29%): [02:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:50:48] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 86944 MB (8% inode=99%): [02:52:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:53:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:06:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:21:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90238 MB (14% inode=99%): [03:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 286182 MB (5% inode=29%): [03:27:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:21:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89290 MB (14% inode=99%): [04:26:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 284689 MB (5% inode=29%): [04:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:52:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:54:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:05:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:21:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89143 MB (14% inode=99%): [05:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 282460 MB (5% inode=29%): [05:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:21:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89053 MB (14% inode=99%): [06:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 285045 MB (5% inode=29%): [06:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:52:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [08:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 281053 MB (5% inode=29%): [08:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88826 MB (14% inode=99%): [08:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [09:06:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:18:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 280491 MB (5% inode=29%): [09:27:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88628 MB (14% inode=99%): [09:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:54:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [10:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:26:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279130 MB (5% inode=28%): [10:27:48] /sql on cassia is WARNING: DISK WARNING - free space: /sql 91098 MB (7% inode=99%): [10:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88415 MB (14% inode=99%): [10:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [11:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 281265 MB (5% inode=29%): [11:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:27:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88182 MB (14% inode=99%): [11:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [12:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:15:58] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2163 [12:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:19:48] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2209 [12:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:26:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 277353 MB (5% inode=28%): [12:27:49] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [12:27:49] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [12:27:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88008 MB (14% inode=99%): [12:39:49] MySQL on z-dat-s6-a is OK: Uptime: 1065 Threads: 6 Questions: 681 Slow queries: 11 Opens: 122 Flush tables: 1 Open tables: 111 Queries per second avg: 0.639 [12:39:49] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2821 [12:40:58] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3622 [12:42:50] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57732 MB (5% inode=99%): [12:43:48] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628 [12:47:48] MySQL slave on z-dat-s6-a is OK: Uptime: 1545 Threads: 7 Questions: 134778 Slow queries: 22 Opens: 231 Flush tables: 1 Open tables: 212 Queries per second avg: 87.234 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1780 [12:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:05:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:26:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 278639 MB (5% inode=28%): [13:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:27:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87791 MB (14% inode=99%): [13:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 4184 MB (20% inode=91%): [13:36:58] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3374 [13:42:59] MySQL slave on z-dat-s7-a is OK: Uptime: 165890 Threads: 5 Questions: 63338237 Slow queries: 5065 Opens: 39431 Flush tables: 1 Open tables: 4317 Queries per second avg: 381.808 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1606 [13:43:48] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4870 [13:51:48] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3428 [13:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:11:48] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 88219 MB (9% inode=99%): [14:12:49] MySQL slave on z-dat-s3-a is OK: Uptime: 94874 Threads: 16 Questions: 111816616 Slow queries: 8753 Opens: 488609 Flush tables: 1 Open tables: 16383 Queries per second avg: 1178.580 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1711 [14:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 275632 MB (5% inode=28%): [14:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:27:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87554 MB (14% inode=99%): [14:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 4088 MB (20% inode=91%): [14:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:06:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 275168 MB (5% inode=28%): [15:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87376 MB (14% inode=99%): [15:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 3988 MB (19% inode=91%): [15:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:54:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 276887 MB (5% inode=28%): [16:28:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:28:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87127 MB (14% inode=99%): [16:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 3874 MB (19% inode=91%): [16:52:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [17:06:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 272949 MB (5% inode=28%): [17:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:27:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86812 MB (14% inode=99%): [17:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 3745 MB (18% inode=91%): [17:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:26:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 275472 MB (5% inode=28%): [18:27:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86633 MB (14% inode=99%): [18:34:59] / on ptolemy is WARNING: DISK WARNING - free space: / 3621 MB (18% inode=91%): [18:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:26:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 274675 MB (5% inode=28%): [19:27:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86446 MB (14% inode=99%): [19:34:59] / on ptolemy is WARNING: DISK WARNING - free space: / 3503 MB (17% inode=91%): [19:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 273692 MB (5% inode=28%): [20:27:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86287 MB (14% inode=99%): [20:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 3388 MB (16% inode=91%): [20:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:05:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:06:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:18:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 272872 MB (5% inode=28%): [21:27:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85835 MB (14% inode=99%): [21:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 3282 MB (16% inode=91%): [21:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:05:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:06:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:24:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 272001 MB (5% inode=28%): [22:27:48] /sql on cassia is WARNING: DISK WARNING - free space: /sql 90264 MB (7% inode=99%): [22:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:27:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85638 MB (14% inode=99%): [22:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 3179 MB (15% inode=91%): [22:52:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:05:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:06:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:18:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:26:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 271211 MB (5% inode=28%): [23:27:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:27:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85436 MB (14% inode=99%): [23:34:58] / on ptolemy is WARNING: DISK WARNING - free space: / 3114 MB (15% inode=91%): [23:52:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:54:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:54:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7