[00:02:37] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:07:10] Load avg. on willow is WARNING: WARNING - load average: 16.27, 16.78, 16.05 [00:08:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:15:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715816.000000 [00:24:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321713 MB (6% inode=55%): [00:34:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:40:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:44:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:45:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:00:18] Free Memory on damiana is WARNING: WARNING - 6.8% (571020 kB) free! [01:02:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:03:18] Free Memory on damiana is OK: OK - 7.9% (663344 kB) free. [01:07:09] Load avg. on willow is WARNING: WARNING - load average: 17.78, 18.14, 17.49 [01:08:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:15:25] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715676.000000 [01:16:18] Load avg. on ortelius is WARNING: WARNING - load average: 29.40, 20.72, 11.63 [01:17:18] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 31.12, 22.77, 12.94 [01:22:18] Load avg. on ortelius is WARNING: WARNING - load average: 16.85, 22.84, 16.09 [01:24:18] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324216 MB (6% inode=55%): [01:35:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:40:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:44:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:45:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:55:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:55:41] Load avg. on ortelius is WARNING: WARNING - load average: 10.81, 12.34, 15.38 [02:02:37] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:07:08] Load avg. on willow is WARNING: WARNING - load average: 20.16, 20.02, 19.12 [02:08:18] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 31.38, 23.91, 19.67 [02:08:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:15:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713259.000000 [02:18:18] Load avg. on ortelius is WARNING: WARNING - load average: 3.85, 16.05, 19.47 [02:19:08] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 79565 MB (8% inode=99%): [02:25:30] Load avg. on ortelius is OK: OK - load average: 2.13, 7.42, 14.67 [02:25:30] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 323899 MB (6% inode=55%): [02:34:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:40:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:44:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:45:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:53:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:02:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:07:08] Load avg. on willow is WARNING: WARNING - load average: 21.07, 19.50, 18.48 [03:08:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:15:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715142.000000 [03:16:08] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.67, 22.99, 20.52 [03:24:18] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321189 MB (6% inode=55%): [03:34:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:40:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:44:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:45:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:54:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:58:08] Load avg. on willow is WARNING: WARNING - load average: 18.12, 19.27, 19.98 [04:02:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.84, 20.11, 20.12 [04:02:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:04:38] Load avg. on willow is WARNING: WARNING - load average: 17.73, 19.25, 19.80 [04:08:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:15:37] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715666.000000 [04:24:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320423 MB (6% inode=55%): [04:34:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [04:40:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:44:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:45:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:51:38] Load avg. on willow is WARNING: WARNING - load average: 19.39, 19.68, 19.76 [05:02:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:08:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:15:37] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 717000.000000 [05:23:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:24:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320673 MB (6% inode=55%): [05:34:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:40:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:44:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:45:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:51:37] Load avg. on willow is WARNING: WARNING - load average: 19.12, 19.14, 18.80 [06:02:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:02:57] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128661 MB (10% inode=99%): [06:08:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:11:57] /sql on cassia is OK: DISK OK - free space: /sql 130698 MB (11% inode=99%): [06:15:37] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 716634.000000 [06:19:57] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128878 MB (10% inode=99%): [06:23:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:24:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319726 MB (6% inode=55%): [06:34:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:40:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:42:57] /sql on cassia is OK: DISK OK - free space: /sql 131735 MB (11% inode=99%): [06:44:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:45:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:51:37] Load avg. on willow is WARNING: WARNING - load average: 16.62, 16.04, 16.68 [07:02:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:08:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:12:38] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319480 MB (5% inode=55%): [07:13:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319554 MB (6% inode=55%): [07:14:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319240 MB (5% inode=55%): [07:15:37] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 716822.000000 [07:20:37] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1580 MB (7% inode=99%): [07:23:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:25:37] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.34, 19.15, 17.03 [07:26:38] Load avg. on willow is WARNING: WARNING - load average: 21.23, 18.51, 16.94 [07:34:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [07:41:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:45:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:02:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:08:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:14:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318279 MB (5% inode=55%): [08:15:37] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 717486.000000 [08:20:37] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1636 MB (8% inode=99%): [08:23:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:26:37] Load avg. on willow is WARNING: WARNING - load average: 16.04, 15.56, 15.72 [08:34:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [08:41:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:44:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:45:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:54:37] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2418 MB (11% inode=99%): [08:56:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 2014 MB (9% inode=99%): [09:02:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:06:54] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (407968 kB) free! [09:08:54] Free Memory on damiana is WARNING: WARNING - 5.7% (479024 kB) free! [09:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:14:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 316471 MB (5% inode=55%): [09:16:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 716907.000000 [09:18:54] Free Memory on damiana is OK: OK - 7.1% (596368 kB) free. [09:27:34] Load avg. on willow is WARNING: WARNING - load average: 15.26, 15.98, 15.95 [09:33:55] Free Memory on damiana is WARNING: WARNING - 6.1% (509828 kB) free! [09:34:54] Free Memory on damiana is OK: OK - 7.4% (620512 kB) free. [09:34:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:41:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:44:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:46:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:56:43] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1356 MB (6% inode=99%): [10:02:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:08:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317029 MB (5% inode=55%): [10:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 716093.000000 [10:27:34] Load avg. on willow is WARNING: WARNING - load average: 14.89, 15.71, 16.18 [10:34:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:44:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:46:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:56:44] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1663 MB (8% inode=99%): [11:02:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:03:54] Free Memory on damiana is WARNING: WARNING - 6.3% (531768 kB) free! [11:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315877 MB (5% inode=55%): [11:16:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715487.000000 [11:17:54] Free Memory on damiana is OK: OK - 7.2% (601048 kB) free. [11:20:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.88, 21.20, 18.61 [11:21:34] Load avg. on willow is WARNING: WARNING - load average: 21.59, 20.38, 18.50 [11:31:54] Free Memory on damiana is WARNING: WARNING - 6.2% (516864 kB) free! [11:33:59] Free Memory on damiana is OK: OK - 7.5% (631756 kB) free. [11:34:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:45:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:46:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:56:44] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1391 MB (7% inode=99%): [12:02:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315181 MB (5% inode=55%): [12:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713357.000000 [12:20:54] Free Memory on damiana is WARNING: WARNING - 6.0% (503976 kB) free! [12:21:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.46, 21.42, 20.13 [12:22:34] Load avg. on willow is WARNING: WARNING - load average: 17.93, 20.35, 19.83 [12:27:54] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (350756 kB) free! [12:29:54] Free Memory on damiana is WARNING: WARNING - 6.3% (529928 kB) free! [12:32:44] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2414 MB (11% inode=99%): [12:33:44] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 2245 MB (10% inode=99%): [12:34:44] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2361 MB (11% inode=99%): [12:34:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:42:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:45:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:46:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:02:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:03:54] Free Memory on damiana is WARNING: WARNING - 6.3% (524092 kB) free! [13:07:54] Free Memory on damiana is OK: OK - 7.3% (615312 kB) free. [13:08:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:09:44] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1872 MB (9% inode=99%): [13:12:44] /tmp on ortelius is OK: DISK OK - free space: /tmp 17629 MB (88% inode=99%): [13:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 314622 MB (5% inode=55%): [13:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 712828.000000 [13:22:33] Load avg. on willow is WARNING: WARNING - load average: 17.32, 18.39, 18.16 [13:26:54] Free Memory on damiana is WARNING: WARNING - 6.0% (499248 kB) free! [13:34:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:45:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:46:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 44.06, 23.10, 19.66 [14:01:33] Load avg. on willow is WARNING: WARNING - load average: 28.21, 22.36, 19.61 [14:02:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:05:54] Free Memory on damiana is WARNING: WARNING - 5.4% (455416 kB) free! [14:08:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:10:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.15, 20.40, 19.16 [14:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 313925 MB (5% inode=55%): [14:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713191.000000 [14:16:54] Free Memory on damiana is OK: OK - 7.1% (598628 kB) free. [14:19:44] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 79518 MB (8% inode=99%): [14:30:34] Load avg. on willow is WARNING: WARNING - load average: 29.48, 19.14, 18.08 [14:33:54] Free Memory on damiana is WARNING: WARNING - 6.7% (558208 kB) free! [14:34:54] Free Memory on damiana is OK: OK - 7.3% (611592 kB) free. [14:34:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:35:21] !Silke_WMDE_ [14:35:51] Hi Coren [14:37:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 127846 MB (10% inode=99%): [14:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:45:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:46:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:48:04] /sql on cassia is OK: DISK OK - free space: /sql 133996 MB (11% inode=99%): [14:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 40.60, 22.80, 19.38 [15:01:33] Load avg. on willow is WARNING: WARNING - load average: 28.05, 22.48, 19.49 [15:02:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 313288 MB (5% inode=55%): [15:15:54] Free Memory on damiana is WARNING: WARNING - 6.1% (515152 kB) free! [15:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715161.000000 [15:26:53] Free Memory on damiana is OK: OK - 7.2% (599408 kB) free. [15:34:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:45:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:46:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:59:54] Free Memory on damiana is WARNING: WARNING - 5.6% (470736 kB) free! [16:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.74, 20.07, 18.41 [16:01:34] Load avg. on willow is WARNING: WARNING - load average: 23.43, 19.82, 18.43 [16:02:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:11:54] Free Memory on damiana is OK: OK - 7.1% (598940 kB) free. [16:14:48] Oh noes! WMDE 'asplode! [16:14:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 312487 MB (5% inode=55%): [16:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 714945.000000 [16:22:54] Free Memory on damiana is WARNING: WARNING - 6.1% (514512 kB) free! [16:27:54] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (338476 kB) free! [16:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:45:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:47:08] @replag [16:47:08] Alchimista: s1-rr-a-wd: 14s [-0.00 s/s]; s1-user: 39s [-0.00 s/s]; s1-user-wd: 13s [-0.00 s/s]; s2-user-c: 1w 1d 6h 6m 49s [-0.11 s/s]; s3-user: 15s [-0.00 s/s]; s4-user-wd: 14s [-0.00 s/s]; s5-user-c: 1w 1d 6h 6m 48s [-0.11 s/s]; s6-user-wd: 10s [-0.00 s/s] [16:47:09] Alchimista: s7-user-wd: 12s [-0.00 s/s] [16:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:57:34] Load avg. on willow is OK: OK - load average: 13.64, 14.39, 15.00 [17:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:06:34] Load avg. on willow is WARNING: WARNING - load average: 14.41, 15.46, 15.50 [17:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:11:05] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129615 MB (10% inode=99%): [17:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 311950 MB (5% inode=55%): [17:16:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 712513.000000 [17:28:34] Load avg. on willow is OK: OK - load average: 13.31, 14.39, 14.99 [17:31:17] hello all [17:31:20] @replag [17:31:20] DaBPunkt: s1-rr-a-wd: 10s [-0.00 s/s]; s1-user: 26s [-0.00 s/s]; s2-user: 14s [-0.03 s/s]; s2-user-c: 1w 1d 5h 55m 30s [-0.26 s/s]; s3-user: 1m 18s [+0.02 s/s]; s4-user-wd: 10s [-0.00 s/s]; s5-user-c: 1w 1d 5h 55m 30s [-0.26 s/s]; s7-user-wd: 10s [-0.00 s/s] [17:33:15] hello DaBPunkt. seems to be a problem on s2 [17:33:43] on ptwiki_p.recentchanges, at least some data prior to 2013 seems missing [17:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [17:36:33] Load avg. on willow is WARNING: WARNING - load average: 15.35, 15.57, 15.43 [17:39:32] Alchimista: errors in the recent-changes-tables wozld be not THAT bad – but s2 is planned for an import anyway in near future [17:40:04] /sql on cassia is OK: DISK OK - free space: /sql 130129 MB (11% inode=99%): [17:42:05] DaBPunkt: how near? can you risk on a prediction? one month? just to have an ideia on when it will be avaiable again. [17:42:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:42:52] a month should be enough [17:43:19] ok then. i'll pay attention to the mlist [17:45:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:47:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129792 MB (10% inode=99%): [17:53:34] Load avg. on willow is OK: OK - load average: 12.80, 14.25, 14.94 [17:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:02:53] Free Memory on damiana is WARNING: WARNING - 5.8% (485660 kB) free! [18:03:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:04:04] /sql on cassia is OK: DISK OK - free space: /sql 130321 MB (11% inode=99%): [18:06:34] Load avg. on willow is WARNING: WARNING - load average: 20.45, 17.11, 15.90 [18:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:10:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.34, 22.22, 18.13 [18:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 311452 MB (5% inode=55%): [18:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 710990.000000 [18:20:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129413 MB (10% inode=99%): [18:20:54] Free Memory on damiana is OK: OK - 7.1% (594380 kB) free. [18:23:05] /sql on cassia is OK: DISK OK - free space: /sql 130165 MB (11% inode=99%): [18:30:34] Load avg. on willow is WARNING: WARNING - load average: 23.22, 16.86, 17.04 [18:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:45:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:49:05] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129720 MB (10% inode=99%): [18:52:54] Free Memory on damiana is WARNING: WARNING - 5.6% (470072 kB) free! [18:53:34] Load avg. on willow is OK: OK - load average: 13.47, 14.20, 14.99 [18:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:02:54] Free Memory on damiana is OK: OK - 7.1% (591732 kB) free. [19:03:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:06:34] Load avg. on willow is WARNING: WARNING - load average: 14.39, 15.50, 15.31 [19:08:04] /sql on cassia is OK: DISK OK - free space: /sql 130234 MB (11% inode=99%): [19:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:09:34] Load avg. on willow is OK: OK - load average: 13.46, 14.44, 14.91 [19:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 310679 MB (5% inode=55%): [19:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 709737.000000 [19:17:54] Free Memory on damiana is WARNING: WARNING - 6.6% (551204 kB) free! [19:27:55] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (340752 kB) free! [19:29:54] Free Memory on damiana is WARNING: WARNING - 5.6% (470332 kB) free! [19:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:45:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:47:34] Load avg. on willow is WARNING: WARNING - load average: 18.04, 18.18, 17.47 [19:49:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 130035 MB (10% inode=99%): [19:54:04] /sql on cassia is OK: DISK OK - free space: /sql 131329 MB (11% inode=99%): [19:54:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.39, 20.55, 18.30 [20:01:34] Load avg. on willow is WARNING: WARNING - load average: 23.45, 20.16, 18.31 [20:03:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:06:54] Free Memory on damiana is WARNING: WARNING - 6.3% (524892 kB) free! [20:09:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:13:54] Free Memory on damiana is OK: OK - 7.2% (606988 kB) free. [20:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 309920 MB (5% inode=55%): [20:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 709252.000000 [20:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:44:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129853 MB (10% inode=99%): [20:46:04] /sql on cassia is OK: DISK OK - free space: /sql 130809 MB (11% inode=99%): [20:46:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:58:54] Free Memory on damiana is WARNING: WARNING - 5.4% (455776 kB) free! [21:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.17, 19.33, 17.94 [21:01:34] Load avg. on willow is WARNING: WARNING - load average: 22.15, 19.02, 17.92 [21:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:13:54] Free Memory on damiana is OK: OK - 7.1% (594796 kB) free. [21:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 309279 MB (5% inode=55%): [21:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 708889.000000 [21:20:54] Free Memory on damiana is WARNING: WARNING - 6.7% (565288 kB) free! [21:21:48] hi, I'm running a job through cronie as */10 * * * * qcronsub -l arch=lx -l h_rt=INFINITY -l virtual_free=256M $HOME/wsbot/wsocrdaemon [21:22:15] but qstat show it starts on solaris (willow) [21:22:27] 1248280 0.50002 wsocrdaemo phe r 01/30/2013 20:59:55 longrun-sol@willow.toolserver. 1 [21:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:27:54] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (370596 kB) free! [21:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:46:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:47:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 43.22, 22.58, 18.54 [22:01:34] Load avg. on willow is WARNING: WARNING - load average: 27.00, 21.72, 18.50 [22:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 308898 MB (5% inode=55%): [22:15:53] re [22:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 710652.000000 [22:17:08] @replag [22:17:08] DaBPunkt: s1-user-wd: 12s [-0.00 s/s]; s2-user-c: 1w 1d 5h 24m 40s [-0.11 s/s]; s2-user-wd: 1m 31s [+0.00 s/s]; s3-user: 11s [-0.00 s/s]; s5-user-c: 1w 1d 5h 24m 38s [-0.11 s/s]; s5-user-wd: 1m 31s [+0.00 s/s]; s7-user-wd: 11s [+0.00 s/s] [22:21:41] I solved my trouble by submiting my cron job on submit.toolserver.org rather than on willow [22:22:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129412 MB (10% inode=99%): [22:23:34] DaBPunkt, FYI I'd trouble with a cron job submited on willow [22:23:43] I was running a job through cronie on willow as */10 * * * * qcronsub -l arch=lx -l h_rt=INFINITY -l virtual_free=256M $HOME/wsbot/wsocrdaemon [22:23:43] but qstat showed it started on solaris (willow) [22:23:43] 1248280 0.50002 wsocrdaemo phe r 01/30/2013 20:59:55 longrun-sol@willow.toolserver. 1 [22:24:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:28:04] /sql on cassia is OK: DISK OK - free space: /sql 130734 MB (11% inode=99%): [22:29:31] phe: that's strange. I will look where the problem might be [22:31:30] the complete output of qstat -j showed SGE tried only solaris box [22:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:36:02] phe: ok, I can repeat it. [22:40:33] looks like it can not shedule jobs with a very long runtime on linux. I will speak with Merlissimo about it [22:42:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:43:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128749 MB (10% inode=99%): [22:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:47:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:55:55] Free Memory on damiana is WARNING: WARNING - 6.0% (505768 kB) free! [23:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 38.48, 21.70, 18.45 [23:01:33] Load avg. on willow is WARNING: WARNING - load average: 25.05, 20.86, 18.36 [23:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:08:54] Free Memory on damiana is OK: OK - 7.2% (603584 kB) free. [23:09:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 307882 MB (5% inode=55%): [23:16:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713195.000000 [23:33:54] Free Memory on damiana is WARNING: WARNING - 6.7% (559380 kB) free! [23:34:54] Free Memory on damiana is OK: OK - 7.3% (612408 kB) free. [23:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:44:34] Load avg. on willow is OK: OK - load average: 13.38, 14.33, 14.98 [23:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:47:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL