[00:00:03] DaB. * Re: [Toolserver-l] Adding more roots [00:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:10:28] Free Memory on damiana is WARNING: WARNING - 6.1% (513668 kB) free! [00:17:28] Free Memory on damiana is OK: OK - 7.1% (596328 kB) free. [00:34:27] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (337980 kB) free! [00:38:40] nacht ts [00:39:27] Free Memory on damiana is WARNING: WARNING - 5.1% (428984 kB) free! [00:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 187810 MB (3% inode=52%): [00:57:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:57:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:57:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1208783.000000 [00:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102958 MB (16% inode=99%): [00:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.36, 23.35, 23.49 [00:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:04] Tim Landscheidt * Re: [Toolserver-l] Adding more roots [01:13:27] Free Memory on damiana is OK: OK - 7.1% (594028 kB) free. [01:16:06] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:31:27] Free Memory on damiana is WARNING: WARNING - 5.4% (448448 kB) free! [01:39:37] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2181.000000 [01:46:27] Free Memory on damiana is OK: OK - 7.1% (593016 kB) free. [01:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 187686 MB (3% inode=52%): [01:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1210452.000000 [01:58:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100930 MB (16% inode=99%): [01:58:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.39, 24.44, 24.34 [01:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [02:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:22:17] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 75856 MB (7% inode=99%): [02:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 182806 MB (3% inode=52%): [02:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1212567.000000 [02:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102593 MB (16% inode=99%): [02:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.10, 23.71, 24.21 [02:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:03:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7221.000000 [03:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 154319 MB (2% inode=52%): [03:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1214504.000000 [03:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102519 MB (16% inode=99%): [03:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.97, 24.88, 25.45 [03:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:03:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10820.000000 [04:04:26] Free Memory on damiana is WARNING: WARNING - 5.1% (423652 kB) free! [04:10:27] Free Memory on damiana is OK: OK - 8.5% (708104 kB) free. [04:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 130158 MB (2% inode=52%): [04:57:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1216971.000000 [04:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102433 MB (16% inode=99%): [04:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.68, 27.04, 27.11 [04:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:59:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14420.000000 [05:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 103305 MB (1% inode=52%): [05:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:57:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:57:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:57:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1218790.000000 [05:57:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102366 MB (16% inode=99%): [05:57:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.41, 26.13, 27.42 [05:58:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:58:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:03:36] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:03:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18021.000000 [06:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:17:27] Free Memory on damiana is WARNING: WARNING - 6.3% (528724 kB) free! [06:28:27] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419556 kB) free! [06:29:27] Free Memory on damiana is WARNING: WARNING - 6.3% (524724 kB) free! [06:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 79752 MB (1% inode=52%): [06:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1221399.000000 [06:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102259 MB (16% inode=99%): [06:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.84, 26.18, 26.63 [06:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21620.000000 [07:07:28] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (399860 kB) free! [07:08:27] Free Memory on damiana is WARNING: WARNING - 5.3% (444624 kB) free! [07:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:17:27] Free Memory on damiana is OK: OK - 7.7% (644044 kB) free. [07:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 166986 MB (3% inode=52%): [07:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:57:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:57:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1223348.000000 [07:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102118 MB (16% inode=99%): [07:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.09, 23.54, 24.10 [07:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25281.000000 [08:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:33:27] Free Memory on damiana is WARNING: WARNING - 6.9% (574448 kB) free! [08:34:27] Free Memory on damiana is OK: OK - 7.5% (626916 kB) free. [08:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 166405 MB (3% inode=52%): [08:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1225768.000000 [08:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101871 MB (16% inode=99%): [08:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.29, 23.81, 24.11 [08:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:00:28] Free Memory on damiana is WARNING: WARNING - 5.2% (432664 kB) free! [09:02:28] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (398424 kB) free! [09:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:27] Free Memory on damiana is WARNING: WARNING - 5.1% (424264 kB) free! [09:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28880.000000 [09:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:19:27] Free Memory on damiana is OK: OK - 7.1% (593896 kB) free. [09:33:27] Free Memory on damiana is WARNING: WARNING - 6.8% (569480 kB) free! [09:35:27] Free Memory on damiana is OK: OK - 7.5% (630092 kB) free. [09:51:28] /sql on cassia is WARNING: DISK WARNING - free space: /sql 123398 MB (10% inode=99%): [09:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 165456 MB (3% inode=52%): [09:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:57:27] Free Memory on damiana is WARNING: WARNING - 5.2% (439104 kB) free! [09:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:57:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:57:46] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1226668.000000 [09:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101635 MB (16% inode=99%): [09:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.41, 23.57, 24.13 [09:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32482.000000 [10:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:26:37] SSH on thyme is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:33:26] Free Memory on damiana is WARNING: WARNING - 5.8% (484416 kB) free! [10:33:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 164763 MB (3% inode=52%): [10:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1229194.000000 [10:57:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101411 MB (16% inode=99%): [10:57:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.10, 23.21, 23.47 [10:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:58:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:01:32] Free Memory on damiana is OK: OK - 7.1% (596720 kB) free. [11:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36074.000000 [11:08:27] Free Memory on damiana is WARNING: WARNING - 6.0% (504972 kB) free! [11:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:23:32] Free Memory on damiana is OK: OK - 7.2% (600868 kB) free. [11:33:27] Free Memory on damiana is WARNING: WARNING - 6.7% (558792 kB) free! [11:33:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:30] Free Memory on damiana is OK: OK - 7.2% (607516 kB) free. [11:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 163704 MB (3% inode=52%): [11:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1230432.000000 [11:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101159 MB (16% inode=99%): [11:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.29, 22.78, 23.08 [11:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39673.000000 [12:08:27] Free Memory on damiana is WARNING: WARNING - 6.4% (540192 kB) free! [12:16:06] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:18:26] Free Memory on damiana is OK: OK - 7.1% (591348 kB) free. [12:25:27] Free Memory on damiana is WARNING: WARNING - 5.7% (475792 kB) free! [12:33:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:27] Free Memory on damiana is WARNING: WARNING - 7.0% (589788 kB) free! [12:48:27] Free Memory on damiana is OK: OK - 7.3% (609652 kB) free. [12:57:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 157890 MB (2% inode=52%): [12:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:57:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:57:28] Free Memory on damiana is WARNING: WARNING - 6.2% (522672 kB) free! [12:57:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:57:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:57:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1231388.000000 [12:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100932 MB (16% inode=99%): [12:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.71, 23.50, 24.41 [12:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43273.000000 [13:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:16:27] Free Memory on damiana is OK: OK - 7.1% (598552 kB) free. [13:31:27] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (375504 kB) free! [13:33:36] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:27] Free Memory on damiana is WARNING: WARNING - 5.2% (432852 kB) free! [13:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 161628 MB (3% inode=52%): [13:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:57:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1233646.000000 [13:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100694 MB (16% inode=99%): [13:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.03, 23.24, 23.44 [13:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46873.000000 [14:05:27] Free Memory on damiana is OK: OK - 7.1% (597628 kB) free. [14:15:27] SSH on thyme is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:18:23] hello all [14:18:28] @replag [14:18:28] DaBPunkt: s1-rr-a-wd: 18s [-0.00 s/s]; s1-user: 13h 15m 8s [+0.09 s/s]; s1-user-wd: 23s [-0.00 s/s]; s2-user: 41m 22s [+0.01 s/s]; s2-user-c: 2w 6h 59m 11s [+0.58 s/s]; s2-user-wd: 24s [-0.01 s/s]; s3-user: 17s [+0.00 s/s]; s4-user-wd: 26s [-0.00 s/s] [14:18:29] DaBPunkt: s5-user-c: 2w 6h 59m 11s [+0.58 s/s]; s5-user-wd: 24s [-0.01 s/s]; s6-user-wd: 19s [-0.00 s/s]; s7-user-wd: 24s [-0.00 s/s] [14:21:47] Load avg. on thyme is CRITICAL: CRITICAL - load average: 2.67, 74.10, 57.82 [14:22:17] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 73044 MB (7% inode=99%): [14:33:27] Free Memory on damiana is WARNING: WARNING - 6.9% (577676 kB) free! [14:34:27] Free Memory on damiana is OK: OK - 7.7% (644876 kB) free. [14:38:47] Load avg. on thyme is WARNING: WARNING - load average: 0.42, 3.04, 19.15 [14:42:47] Load avg. on thyme is OK: OK - load average: 0.58, 1.68, 14.84 [14:51:27] Free Memory on damiana is WARNING: WARNING - 6.1% (509028 kB) free! [14:57:16] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 160979 MB (3% inode=52%): [14:57:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:57:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1236223.000000 [14:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100443 MB (16% inode=99%): [14:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.43, 23.07, 23.22 [14:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50473.000000 [15:16:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:27:27] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (410804 kB) free! [15:35:27] Free Memory on damiana is WARNING: WARNING - 5.6% (470440 kB) free! [15:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:36:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52082 [15:36:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:46:12] Free Memory on damiana is OK: OK - 7.2% (607332 kB) free. [16:36:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 159961 MB (3% inode=52%): [16:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:36:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:36:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:36:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50191.000000 [16:36:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:36:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240911.000000 [16:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 100064 MB (16% inode=99%): [16:36:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.82, 23.61, 24.34 [16:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:36:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50179 [16:36:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:48:12] Free Memory on damiana is WARNING: WARNING - 6.1% (511400 kB) free! [17:17:12] Free Memory on damiana is OK: OK - 7.1% (594588 kB) free. [17:23:03] Dr. Trigon * Re: [Toolserver-l] Cron on submit [17:26:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (363512 kB) free! [17:34:12] Free Memory on damiana is WARNING: WARNING - 5.1% (426648 kB) free! [17:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 159555 MB (2% inode=52%): [17:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:36:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:36:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:36:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46240.000000 [17:36:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:36:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240227.000000 [17:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 99816 MB (16% inode=99%): [17:36:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.34, 22.19, 22.66 [17:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:36:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46174 [17:36:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:43:12] Free Memory on damiana is OK: OK - 7.2% (600656 kB) free. [18:08:12] Free Memory on damiana is WARNING: WARNING - 7.0% (584460 kB) free! [18:09:12] Free Memory on damiana is OK: OK - 7.1% (591856 kB) free. [18:20:52] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1952.000000 [18:26:12] Free Memory on damiana is WARNING: WARNING - 6.8% (567116 kB) free! [18:32:42] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1932.000000 [18:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 158946 MB (2% inode=52%): [18:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:36:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:36:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:36:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42626.000000 [18:36:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:36:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240632.000000 [18:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 99560 MB (16% inode=99%): [18:36:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.14, 23.71, 23.48 [18:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:36:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:36:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42565 [18:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:39:12] Free Memory on damiana is OK: OK - 7.1% (593772 kB) free. [18:48:12] Free Memory on damiana is WARNING: WARNING - 6.7% (564336 kB) free! [18:51:42] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1904.000000 [18:52:22] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1917.000000 [18:53:42] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1955.000000 [18:54:22] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1967.000000 [19:08:12] Free Memory on damiana is OK: OK - 7.3% (612456 kB) free. [19:20:52] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3258.000000 [19:26:12] Free Memory on damiana is WARNING: WARNING - 6.9% (574640 kB) free! [19:32:43] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3246.000000 [19:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 158338 MB (2% inode=52%): [19:36:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:36:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:36:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:36:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37945.000000 [19:36:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:36:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240445.000000 [19:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 99273 MB (16% inode=99%): [19:36:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.80, 22.80, 22.78 [19:36:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [19:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:36:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37917 [19:36:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:46:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [19:51:12] Free Memory on damiana is OK: OK - 7.4% (617344 kB) free. [19:51:42] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3256.000000 [19:52:22] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3268.000000 [19:53:42] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3302.000000 [19:54:22] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3317.000000 [20:07:12] Free Memory on damiana is WARNING: WARNING - 5.8% (484188 kB) free! [20:09:32] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [20:09:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [20:09:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [20:10:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3621.000000 [20:14:12] Free Memory on damiana is OK: OK - 7.4% (620128 kB) free. [20:29:12] Free Memory on damiana is WARNING: WARNING - 6.0% (504992 kB) free! [20:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 157294 MB (2% inode=52%): [20:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:36:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:36:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:36:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34466.000000 [20:36:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:36:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1239892.000000 [20:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98886 MB (16% inode=99%): [20:36:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.68, 22.61, 23.60 [20:36:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5018.000000 [20:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:36:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34423 [20:36:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:46:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5039.000000 [21:09:31] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4847.000000 [21:09:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4854.000000 [21:09:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4852.000000 [21:10:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4862.000000 [21:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 156824 MB (2% inode=52%): [21:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:36:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:36:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:36:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30907.000000 [21:36:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:36:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240763.000000 [21:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98510 MB (16% inode=99%): [21:36:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.45, 23.61, 22.87 [21:36:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6380.000000 [21:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:36:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30885 [21:36:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:46:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6363.000000 [21:52:12] /sql on cassia is WARNING: DISK WARNING - free space: /sql 124583 MB (10% inode=99%): [22:09:32] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5670.000000 [22:09:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5680.000000 [22:09:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5682.000000 [22:10:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5729.000000 [22:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 156340 MB (2% inode=52%): [22:36:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:36:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:36:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:36:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29592.000000 [22:36:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:36:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240253.000000 [22:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98155 MB (16% inode=99%): [22:36:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.23, 22.25, 22.76 [22:36:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7433.000000 [22:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:36:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29549 [22:36:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:46:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7158.000000 [23:09:32] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6654.000000 [23:09:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6886.000000 [23:09:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6662.000000 [23:10:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6760.000000 [23:29:03] Sumana Harihareswara * Re: [Toolserver-l] Sick [23:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 155380 MB (2% inode=52%): [23:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:36:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:36:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:36:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26278.000000 [23:36:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:36:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240169.000000 [23:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97893 MB (16% inode=99%): [23:36:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.29, 22.27, 22.28 [23:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:36:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8456.000000 [23:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:36:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26251 [23:36:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:46:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8298.000000