[00:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [00:08:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319269 MB (6% inode=31%): [00:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103916 MB (17% inode=99%): [00:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:30:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318247 MB (5% inode=31%): [00:37:52] / on ptolemy is WARNING: DISK WARNING - free space: / 3410 MB (17% inode=91%): [00:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:53:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.93, 23.43, 23.56 [00:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:57:42] / on willow is WARNING: DISK WARNING - free space: / 20409 MB (19% inode=98%): [00:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20409 MB (19% inode=98%): [01:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [01:13:12] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2033 [01:13:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2035.000000 [01:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103662 MB (17% inode=99%): [01:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:30:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317657 MB (5% inode=31%): [01:37:52] / on ptolemy is WARNING: DISK WARNING - free space: / 3368 MB (16% inode=91%): [01:41:52] / on ptolemy is OK: DISK OK - free space: / 7557 MB (37% inode=91%): [01:49:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.85, 25.00, 24.89 [01:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:57:42] / on willow is WARNING: DISK WARNING - free space: / 20324 MB (19% inode=98%): [01:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 20324 MB (19% inode=98%): [02:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:01:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:13:12] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3250 [02:13:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3249.000000 [02:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104289 MB (17% inode=99%): [02:22:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321190 MB (6% inode=32%): [02:24:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:31:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3622 [02:31:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3622.000000 [02:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:50:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 88675 MB (9% inode=99%): [02:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.93, 24.64, 25.04 [02:54:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:57:42] / on willow is WARNING: DISK WARNING - free space: / 20242 MB (19% inode=98%): [02:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20242 MB (19% inode=98%): [03:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [03:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:20:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318257 MB (5% inode=31%): [03:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103422 MB (17% inode=99%): [03:24:19] nacht ts [03:31:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4862 [03:31:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4863.000000 [03:49:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:53:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.75, 24.60, 25.04 [03:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:54:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:57:42] / on willow is WARNING: DISK WARNING - free space: / 20157 MB (19% inode=98%): [03:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20157 MB (19% inode=98%): [04:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [04:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:20:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 316254 MB (5% inode=31%): [04:20:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103155 MB (16% inode=99%): [04:31:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6332 [04:31:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6335.000000 [04:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.77, 25.37, 26.27 [04:54:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:57:42] / on willow is WARNING: DISK WARNING - free space: / 20083 MB (19% inode=98%): [04:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 20083 MB (19% inode=98%): [05:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:01:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [05:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:20:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 314376 MB (5% inode=31%): [05:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102950 MB (16% inode=99%): [05:31:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6622 [05:31:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6621.000000 [05:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:54:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.43, 25.76, 25.51 [05:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:55:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:57:43] / on willow is WARNING: DISK WARNING - free space: / 19999 MB (19% inode=98%): [05:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19999 MB (19% inode=98%): [06:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:01:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:18:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:20:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317028 MB (5% inode=31%): [06:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102869 MB (16% inode=99%): [06:31:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5789 [06:31:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5787.000000 [06:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.71, 25.50, 25.38 [06:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:56:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:57:43] / on willow is WARNING: DISK WARNING - free space: / 19912 MB (19% inode=98%): [06:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19912 MB (19% inode=98%): [07:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:01:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:20:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315401 MB (5% inode=31%): [07:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102749 MB (16% inode=99%): [07:23:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:24:12] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3556 [07:24:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3546.000000 [07:47:12] MySQL slave on rosemary is OK: Uptime: 2196364 Threads: 9 Questions: 1501898021 Slow queries: 459584 Opens: 141573 Flush tables: 1 Open tables: 3706 Queries per second avg: 683.811 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1740 [07:47:22] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1729.000000 [07:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.05, 23.19, 24.20 [07:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:56:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:57:41] / on willow is WARNING: DISK WARNING - free space: / 19854 MB (18% inode=98%): [07:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19854 MB (18% inode=98%): [08:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:02:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [08:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:20:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 313658 MB (5% inode=31%): [08:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102601 MB (16% inode=99%): [08:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.07, 26.80, 27.39 [08:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:56:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:57:42] / on willow is WARNING: DISK WARNING - free space: / 19801 MB (18% inode=98%): [08:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19801 MB (18% inode=98%): [09:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:02:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [09:13:11] way to be verbose tsnag [09:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:20:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 311887 MB (5% inode=31%): [09:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102435 MB (16% inode=99%): [09:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.60, 24.91, 25.64 [09:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:56:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:57:42] / on willow is WARNING: DISK WARNING - free space: / 19749 MB (18% inode=98%): [09:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19749 MB (18% inode=98%): [10:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:02:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [10:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:20:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 311212 MB (5% inode=31%): [10:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102252 MB (16% inode=99%): [10:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:27:42] /sql on cassia is WARNING: DISK WARNING - free space: /sql 94408 MB (7% inode=99%): [10:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.20, 25.32, 26.32 [10:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:56:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:57:42] / on willow is WARNING: DISK WARNING - free space: / 19700 MB (18% inode=98%): [10:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19700 MB (18% inode=98%): [11:00:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:02:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:20:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 313481 MB (5% inode=31%): [11:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102078 MB (16% inode=99%): [11:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:49:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.34, 26.57, 26.37 [11:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:57:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:57:42] / on willow is WARNING: DISK WARNING - free space: / 19655 MB (18% inode=98%): [11:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19655 MB (18% inode=98%): [12:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:02:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101856 MB (16% inode=99%): [12:21:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 312070 MB (5% inode=31%): [12:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.84, 24.12, 25.11 [12:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:57:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:57:42] / on willow is WARNING: DISK WARNING - free space: / 19608 MB (18% inode=98%): [12:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19608 MB (18% inode=98%): [13:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [13:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101591 MB (16% inode=99%): [13:21:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 310851 MB (5% inode=31%): [13:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.35, 24.04, 24.26 [13:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:57:42] / on willow is WARNING: DISK WARNING - free space: / 19563 MB (18% inode=98%): [13:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19563 MB (18% inode=98%): [13:58:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:00:12] hello all [14:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [14:03:41] apmon_: arround? [14:03:57] DaBPunkt: Yes [14:04:38] ok, I install the new mod_tile now [14:04:56] cheers. [14:05:41] apmon_: ok, done. Please check if it's work [14:05:57] acccess shows activity [14:06:13] hi DaBPunkt! thanks for already creating anja's account [14:06:25] apmon_: the error-log is silent :) [14:06:28] DanielK_WMDE_: no problem [14:06:51] Ah, good. It also looks like it is working [14:07:18] :) [14:10:14] DaBPunkt: Any other issues that need addressing from your point of view? [14:10:26] no [14:10:35] good. [14:10:59] I think I have also fixed the expire script now to not change the atime back to 1999 [14:11:43] So hopefully the clean up script will now delete the correct files and not the most commonly used ones [14:11:52] '"GET /tiles/hikebike/13/5939/3955.png HTTP/1.0" 404 229 "-" "LocusD/2.0.1 (Linux; U; Android; en-us)"' is that normal? [14:13:16] http://munin.toolserver.org/OSM/ptolemy/mod_tile_response.html shows that overall, there are a fair number of 404 tiles returned [14:13:32] Those occur, if the tiles aren't cached and don't get rendered in time. [14:13:41] ah ok [14:14:13] apmon_: I will TRY to set-up the new OSM-server next week [14:14:51] I suspect the wrong atime / mtime together with the cache clean up might have made it worse, but that should fix it self fairly soon with the common tiles getting cached again. [14:14:55] Sweet! Thanks [14:15:52] apmon_: Please notice that I gave you the Ctime, not the Mtime (to be honest I do not know the differnce, but I'm sure there is) [14:16:21] ctime is the time the inode was changed. [14:16:28] ((in the past I thought that "c" stand for "created", but the man-page says that this is wrong)) [14:16:47] I.e. the time touch reset the mtime and atime back to 1999 to indicate that the tile is outdated and needs rerendering [14:17:10] mtime is modified time, i.e if the content changes but not the inode, then mtime is updated and ctime is not [14:17:11] apmon_: yes, but the "m" stand for "modified" – and where is the differnce between "changed" and "modified"? [14:17:21] mtime and atime are user modifyable, ctime is not [14:17:48] so the ctime only changes when I move a file arround? [14:18:47] yes, or if the mtime is modified using touch (or utime() as is done by render_expired) [14:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:20:05] so if a file is never moved the chance is high that ctime shows the creation-time? [14:20:22] mtime was supposed to say either when it was created or if it is out of date and needs rerendering. Atime should say when the tile was last served (i.e. if it is commonly used and thus should remain cached) [14:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101357 MB (16% inode=99%): [14:21:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 309800 MB (5% inode=31%): [14:21:25] Sort of. Expect for all of the tiles that get expired (which only runs on the styles default, hikebike and two or three others), as it would but too much stat load if it was done for all styles [14:22:01] It is probably not the best system, but it normally more or less works [14:22:55] and I am not sure what a better way to mark outdated tiles would be [14:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:25:08] apmon_: so let me repeat: If I see a osm-file that, has an old atime → not used for some time. Has an old mtime → is outdated? [14:26:59] The expired tiles are set to a date of 1999 which is before the initiation of the OSM project, so can't represent a "true" time. All other times other than from 1999 should reflect the correct time [14:27:13] thanks for creating the planet-import-complete file [14:27:41] np [14:28:31] mod_tile decides which files are outdated by comparing the mtime of the tile to be served against the mtime of the planet-import-complete file. If the tile is older than the planet-import-complete time, it is sent to the rendering daemon for rerendering [14:29:27] so a cleanup needs 2 runs: 1.: delete everything that has a mitime <1999 because they are outdated and need to re-rendered anyway. 2.: Delete all files with a atime if planet-import-complete is missing, it considers all tiles older than 3 days as outdated, which usually overwelmes the rendering, as it doesn't have the capacity to rerender everything every 3 days [14:30:14] mod_tile should take care of all of the <1999 tiles. You only need to do the atime < xDaysinThePast [14:30:37] but planet-import-complete is less than 5min old now – so EVERYTHING is to re-rendered now?? [14:31:30] yes. But in three days it will be older than without the planet-import-complete [14:32:06] so you re-touch the file from time to time? [14:32:06] you could set the mtime of planet-import-complete into the past to cause less tiles to rerender, but occasionally a full rerender / full cache expiry isn't too bad [14:32:20] yes [14:32:29] ok [14:33:43] When the mtime is set to 1999: Would it be possible to set the time+day+month to the actual time+day+month? Than we could see how long it take to handle the files [14:37:59] Yes, that should be possible. I just need to add that logic to https://github.com/openstreetmap/mod_tile/blob/master/render_expired.c#L567 [14:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:50:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 88662 MB (9% inode=99%): [14:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.51, 25.09, 25.46 [14:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:57:42] / on willow is WARNING: DISK WARNING - free space: / 19514 MB (18% inode=98%): [14:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19514 MB (18% inode=98%): [14:58:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [15:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101148 MB (16% inode=99%): [15:21:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 308080 MB (5% inode=31%): [15:24:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:45:38] in 15min s7 will be r/o [15:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.79, 26.15, 26.63 [15:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:57:43] / on willow is WARNING: DISK WARNING - free space: / 19463 MB (18% inode=98%): [15:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19463 MB (18% inode=98%): [15:58:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100930 MB (16% inode=99%): [16:21:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 306904 MB (5% inode=31%): [16:21:52] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [16:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.21, 26.33, 28.32 [16:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:56:04] Sumana Harihareswara * Re: [Toolserver-l] Amsterdam Hackathon 2013 [16:57:42] / on willow is WARNING: DISK WARNING - free space: / 19420 MB (18% inode=98%): [16:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19420 MB (18% inode=98%): [16:58:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:18:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100695 MB (16% inode=99%): [17:21:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 306086 MB (5% inode=30%): [17:21:52] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [17:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.41, 25.30, 25.45 [17:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:57:42] / on willow is WARNING: DISK WARNING - free space: / 19378 MB (18% inode=98%): [17:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19378 MB (18% inode=98%): [17:59:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:00:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:03:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100524 MB (16% inode=99%): [18:21:52] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [18:22:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 308932 MB (5% inode=31%): [18:30:12] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 46392 MB (8% inode=99%): [18:39:05] stopping s7 now [18:40:12] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 31595 MB (5% inode=99%): [18:41:12] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 381726 MB (75% inode=99%): [18:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:49:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.54, 26.40, 26.79 [18:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:57:43] / on willow is WARNING: DISK WARNING - free space: / 19338 MB (18% inode=98%): [18:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19338 MB (18% inode=98%): [19:00:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100291 MB (16% inode=99%): [19:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:22:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 307940 MB (5% inode=31%): [19:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.35, 25.85, 25.95 [19:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:57:43] / on willow is WARNING: DISK WARNING - free space: / 19296 MB (18% inode=98%): [19:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19296 MB (18% inode=98%): [20:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:01:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100057 MB (16% inode=99%): [20:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [20:22:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 307521 MB (5% inode=31%): [20:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [20:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.12, 25.50, 25.83 [20:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:57:42] / on willow is WARNING: DISK WARNING - free space: / 19257 MB (18% inode=98%): [20:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19257 MB (18% inode=98%): [21:00:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:01:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 99894 MB (16% inode=99%): [21:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [21:22:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 306663 MB (5% inode=31%): [21:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [21:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.49, 25.57, 25.29 [21:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:57:42] / on willow is WARNING: DISK WARNING - free space: / 19217 MB (18% inode=98%): [21:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19217 MB (18% inode=98%): [22:00:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:01:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:03:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 99668 MB (16% inode=99%): [22:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [22:22:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 306041 MB (5% inode=30%): [22:27:43] /sql on cassia is WARNING: DISK WARNING - free space: /sql 92961 MB (7% inode=99%): [22:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [22:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.00, 25.59, 25.36 [22:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:57:42] / on willow is WARNING: DISK WARNING - free space: / 19177 MB (18% inode=98%): [22:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19177 MB (18% inode=98%): [23:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 99454 MB (16% inode=99%): [23:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [23:22:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 304849 MB (5% inode=30%): [23:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [23:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:54:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.85, 24.17, 24.47 [23:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:57:43] / on willow is WARNING: DISK WARNING - free space: / 19139 MB (18% inode=98%): [23:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19139 MB (18% inode=98%):