[00:03:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61793 MB (10% inode=99%): [00:05:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:27:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 191502 MB (3% inode=21%): [00:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:42:02] PostgreSQL on ptolemy is CRITICAL: CRITICAL - no connection to osm_mapnik (FATAL: the database system is in recovery mode [00:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:03:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62624 MB (10% inode=99%): [01:05:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:07:01] PostgreSQL on ptolemy is OK: OK - database osm_mapnik (0 sec.) [01:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:27:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 191485 MB (3% inode=21%): [01:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:34:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:56:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:03:13] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63853 MB (10% inode=99%): [02:05:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:13:02] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 85157 MB (8% inode=99%): [02:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:27:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 190267 MB (3% inode=21%): [02:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:34:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:03:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62117 MB (10% inode=99%): [03:05:40] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:18:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:27:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 189268 MB (3% inode=21%): [03:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:34:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:54:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:03:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61962 MB (10% inode=99%): [04:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:27:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 190136 MB (3% inode=21%): [04:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61899 MB (10% inode=99%): [05:05:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:23:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:24:51] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1840 [05:25:21] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1830.000000 [05:27:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 189499 MB (3% inode=21%): [05:29:21] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1794.000000 [05:29:54] MySQL slave on rosemary is OK: Uptime: 2965718 Threads: 21 Questions: 2052048027 Slow queries: 634497 Opens: 191984 Flush tables: 1 Open tables: 3701 Queries per second avg: 691.922 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1786 [05:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:03:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61834 MB (10% inode=99%): [06:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 189711 MB (3% inode=21%): [06:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:56:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61724 MB (10% inode=99%): [07:05:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:18:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 188838 MB (3% inode=21%): [07:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:34:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61615 MB (10% inode=99%): [08:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:27:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 188164 MB (3% inode=21%): [08:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61545 MB (10% inode=99%): [09:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:18:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:23:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 184839 MB (3% inode=21%): [09:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61569 MB (10% inode=99%): [10:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:27:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 185991 MB (3% inode=21%): [10:28:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 82812 MB (6% inode=99%): [10:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61092 MB (10% inode=99%): [11:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 185370 MB (3% inode=21%): [11:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:34:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:48:02] / on willow is WARNING: DISK WARNING - free space: / 21956 MB (20% inode=98%): [11:48:21] /tmp on willow is WARNING: DISK WARNING - free space: / 21956 MB (20% inode=98%): [11:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60799 MB (9% inode=99%): [12:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:18:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 183923 MB (3% inode=21%): [12:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:47:21] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2873 MB (18% inode=99%): [12:48:02] / on willow is WARNING: DISK WARNING - free space: / 21759 MB (20% inode=98%): [12:48:21] /tmp on willow is WARNING: DISK WARNING - free space: / 21759 MB (20% inode=98%): [12:53:02] hello all [12:53:22] /tmp on ortelius is OK: DISK OK - free space: /tmp 3634 MB (22% inode=99%): [12:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:54:29] @replag [12:54:29] DaBPunkt: s1-rr-a-wd: 18m 28s [+0.00 s/s]; s1-user-wd: 50m 18s [+0.01 s/s]; s2-user-wd: 1h 16m 10s [+0.03 s/s]; s3-user: 30s [-0.00 s/s]; s4-user-wd: 18m 28s [+0.01 s/s]; s5-user-wd: 1h 16m 10s [+0.03 s/s]; s6-user-wd: 41m 15s [+0.01 s/s]; s7-user-wd: 37m 42s [+0.00 s/s] [12:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:03:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60584 MB (9% inode=99%): [13:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [13:07:01] /sql on cassia is OK: DISK OK - free space: /sql 393901 MB (33% inode=99%): [13:11:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [13:13:32] ok guys, the re-import of commons at cassia is running [13:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 182496 MB (3% inode=20%): [13:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:48:01] / on willow is WARNING: DISK WARNING - free space: / 21553 MB (20% inode=98%): [13:48:21] /tmp on willow is WARNING: DISK WARNING - free space: / 21553 MB (20% inode=98%): [13:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:56:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60351 MB (9% inode=99%): [14:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [14:11:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [14:13:01] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 84976 MB (8% inode=99%): [14:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:20:25] In case somebody wonders: I update the wikidata-replication at the moment [14:22:21] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3336 MB (20% inode=99%): [14:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:23:21] /tmp on ortelius is OK: DISK OK - free space: /tmp 3761 MB (22% inode=99%): [14:27:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 181572 MB (3% inode=20%): [14:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:48:01] / on willow is WARNING: DISK WARNING - free space: / 21359 MB (20% inode=98%): [14:48:21] /tmp on willow is WARNING: DISK WARNING - free space: / 21359 MB (20% inode=98%): [14:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60188 MB (9% inode=99%): [15:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:11:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [15:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:21:30] DaBPunkt: can we make qsh working now that we have to use sge? [15:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:22:31] I'm very busy [15:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 181064 MB (3% inode=20%): [15:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:48:01] / on willow is WARNING: DISK WARNING - free space: / 21169 MB (20% inode=98%): [15:48:21] /tmp on willow is WARNING: DISK WARNING - free space: / 21169 MB (20% inode=98%): [15:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59985 MB (9% inode=99%): [16:05:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:06:46] @replag [16:06:46] liangent: s1-rr-a-wd: 2h 23m 51s [+0.65 s/s]; s1-user-wd: 2h 41m 34s [+0.58 s/s]; s2-user: 43m 59s [+0.01 s/s]; s2-user-c: error; s2-user-wd: 3h 35m 15s [+0.72 s/s]; s4-user-wd: 2h 26m 34s [+0.67 s/s]; s5-user-c: error; s5-user-wd: 3h 35m 15s [+0.72 s/s] [16:06:47] liangent: s6-user-wd: 2h 26m 34s [+0.55 s/s]; s7-user-wd: 2h 26m 34s [+0.57 s/s] [16:10:34] DaB. is the one who manages the db replication, right? [16:11:04] yes [16:11:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [16:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:24:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:25:01] @replag [16:25:01] DaBPunkt: s1-rr-a-wd: 2h 21m 22s [-0.14 s/s]; s1-user-wd: 2h 59m 49s [+1.00 s/s]; s2-user: 45m 34s [+0.09 s/s]; s2-user-c: error; s2-user-wd: 3h 53m 30s [+1.00 s/s]; s3-user: 60s [+0.00 s/s]; s4-user-wd: 2h 44m 49s [+1.00 s/s]; s5-user-c: error [16:25:02] DaBPunkt: s5-user-wd: 3h 53m 30s [+1.00 s/s]; s6-user-wd: 2h 44m 49s [+1.00 s/s]; s7-user-wd: 2h 44m 49s [+1.00 s/s] [16:25:33] @replag [16:25:34] DaBPunkt: s1-rr-a-wd: 2h 20m 50s [-0.98 s/s]; s1-user: 11s [-0.01 s/s]; s1-user-wd: 3h 22s [+1.01 s/s]; s2-user: 45m 44s [+0.31 s/s]; s2-user-c: error; s2-user-wd: 3h 54m 3s [+1.01 s/s]; s3-user: 1m 33s [+1.01 s/s]; s4-user-wd: 2h 45m 22s [+1.01 s/s] [16:25:35] DaBPunkt: s5-user-c: error; s5-user-wd: 3h 54m 3s [+1.01 s/s]; s6-user-wd: 2h 45m 22s [+1.01 s/s]; s7-user-wd: 2h 45m 22s [+1.01 s/s] [16:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180700 MB (3% inode=20%): [16:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:35:46] @replag [16:35:46] DaBPunkt: s1-rr-a-wd: 2h 21m 10s [+0.03 s/s]; s1-user-wd: 3h 10m 34s [+1.00 s/s]; s2-user: 47m 43s [+0.19 s/s]; s2-user-c: error; s2-user-wd: 4h 4m 15s [+1.00 s/s]; s4-user-wd: 2h 55m 34s [+1.00 s/s]; s5-user-c: error; s5-user-wd: 4h 4m 15s [+1.00 s/s] [16:35:47] DaBPunkt: s6-user-wd: 2h 55m 34s [+1.00 s/s]; s7-user-wd: 2h 55m 34s [+1.00 s/s] [16:48:01] / on willow is WARNING: DISK WARNING - free space: / 20981 MB (20% inode=98%): [16:48:22] /tmp on willow is WARNING: DISK WARNING - free space: / 20981 MB (20% inode=98%): [16:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:57:01] / on willow is OK: DISK OK - free space: / 34933 MB (33% inode=99%): [16:57:21] /tmp on willow is OK: DISK OK - free space: / 34932 MB (33% inode=99%): [17:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59804 MB (9% inode=99%): [17:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [17:11:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [17:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:25:24] @replag [17:25:27] DaBPunkt: s1-rr-a-wd: 1h 24m 10s [-1.15 s/s]; s1-user-wd: 3h 25m 43s [+0.31 s/s]; s2-user: 45m 6s [-0.05 s/s]; s2-user-c: error; s2-user-wd: 4h 24m 59s [+0.42 s/s]; s3-user: 52s [-0.01 s/s]; s4-user-wd: 3h 12m 34s [+0.34 s/s]; s5-user-c: error [17:25:27] DaBPunkt: s5-user-wd: 4h 25m 0s [+0.42 s/s]; s6-user-wd: 3h 12m 34s [+0.34 s/s]; s7-user-wd: 3h 12m 56s [+0.35 s/s] [17:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180099 MB (3% inode=20%): [17:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:53:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59534 MB (9% inode=99%): [18:05:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [18:11:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [18:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:22:21] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2891 MB (18% inode=99%): [18:22:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:26:21] /tmp on ortelius is OK: DISK OK - free space: /tmp 3468 MB (21% inode=99%): [18:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 179910 MB (3% inode=20%): [18:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:34:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:42:21] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3159 MB (19% inode=99%): [18:53:22] /tmp on ortelius is OK: DISK OK - free space: /tmp 3381 MB (21% inode=99%): [18:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59307 MB (9% inode=99%): [19:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [19:11:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [19:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:21:04] @replag [19:21:06] DaBPunkt: s1-rr-a-wd: 51m 29s [-0.28 s/s]; s1-user-wd: 2h 26m 44s [-0.51 s/s]; s2-user: 14m 44s [-0.26 s/s]; s2-user-c: error; s2-user-wd: 3h 11m 7s [-0.64 s/s]; s3-user: 28s [-0.00 s/s]; s4-user-wd: 2h 11m 39s [-0.53 s/s]; s5-user-c: error [19:21:07] DaBPunkt: s5-user-wd: 3h 11m 9s [-0.64 s/s]; s6-user-wd: 2h 12m 5s [-0.52 s/s]; s7-user-wd: 2h 14m 59s [-0.50 s/s] [19:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180296 MB (3% inode=20%): [19:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:54:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59014 MB (9% inode=99%): [20:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [20:11:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [20:18:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 179546 MB (3% inode=20%): [20:29:21] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 159 MB (0% inode=99%): [20:30:21] /tmp on ortelius is OK: DISK OK - free space: /tmp 15830 MB (96% inode=99%): [20:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:34:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:03:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58771 MB (9% inode=99%): [21:05:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [21:11:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [21:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:27:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 179271 MB (3% inode=20%): [21:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:34:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:56:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58537 MB (9% inode=99%): [22:05:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [22:11:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [22:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 178925 MB (3% inode=20%): [22:32:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:34:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58323 MB (9% inode=99%): [23:05:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:11:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Unknown database commonswiki_p [23:18:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:22:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:27:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 177594 MB (3% inode=20%): [23:32:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:34:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:44:50] @replag [23:44:50] Merlissimo: s1-user-wd: 2h 20s [-0.10 s/s]; s2-user: 14m 49s [+0.00 s/s]; s2-user-c: error; s2-user-wd: 4h 14m 49s [+0.24 s/s]; s3-user: 59s [+0.00 s/s]; s4-user-wd: 31m 43s [-0.38 s/s]; s5-user-c: error; s5-user-wd: 4h 14m 49s [+0.24 s/s] [23:44:51] Merlissimo: s6-user-wd: 1h 1m 14s [-0.27 s/s]; s7-user-wd: 56m 42s [-0.30 s/s] [23:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:56:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7