[00:00:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 39.06, 23.31, 19.88 [00:01:51] Load avg. on willow is WARNING: WARNING - load average: 27.36, 22.86, 19.94 [00:02:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:08:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:09:51] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126553 MB (10% inode=99%): [00:10:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.34, 21.91, 20.43 [00:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 285035 MB (5% inode=55%): [00:17:15] nacht ts [00:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 801494.000000 [00:28:51] Load avg. on willow is WARNING: WARNING - load average: 18.13, 19.29, 19.95 [00:30:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.29, 20.85, 20.42 [00:34:52] Load avg. on willow is WARNING: WARNING - load average: 18.25, 19.43, 19.92 [00:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:36:52] Load avg. on nightshade is WARNING: WARNING - load average: 9.09, 15.37, 12.88 [00:37:52] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 36.02, 21.08, 14.93 [00:38:51] Load avg. on nightshade is WARNING: WARNING - load average: 19.20, 18.96, 14.59 [00:42:51] /sql on cassia is OK: DISK OK - free space: /sql 131903 MB (11% inode=99%): [00:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:49:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:54:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:56:21] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:02:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:03:21] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:08:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 283778 MB (5% inode=55%): [01:15:52] Load avg. on ortelius is WARNING: WARNING - load average: 28.39, 20.08, 12.13 [01:16:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.52, 22.09, 21.55 [01:17:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 805094.000000 [01:19:51] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128946 MB (10% inode=99%): [01:26:51] Load avg. on ortelius is OK: OK - load average: 7.31, 14.55, 14.30 [01:26:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:35:51] /sql on cassia is OK: DISK OK - free space: /sql 131071 MB (11% inode=99%): [01:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:49:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:02:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:06:20] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:06:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:08:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:15:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 283087 MB (5% inode=55%): [02:16:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.00, 21.56, 21.48 [02:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 808693.000000 [02:20:51] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 78736 MB (8% inode=99%): [02:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:49:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:02:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:06:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:08:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:10:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 280840 MB (5% inode=55%): [03:16:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.48, 25.10, 24.40 [03:17:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 812294.000000 [03:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:02:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:06:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129852 MB (10% inode=99%): [04:06:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:15:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279877 MB (5% inode=55%): [04:17:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.50, 26.20, 25.81 [04:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 815912.000000 [04:21:54] /sql on cassia is OK: DISK OK - free space: /sql 131193 MB (11% inode=99%): [04:34:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129237 MB (10% inode=99%): [04:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:08:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:12:20] is SGE broken somehow? [05:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 280792 MB (5% inode=55%): [05:17:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.18, 26.89, 26.29 [05:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 819513.000000 [05:23:52] /sql on cassia is OK: DISK OK - free space: /sql 130973 MB (11% inode=99%): [05:30:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 125855 MB (10% inode=99%): [05:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:48:53] /sql on cassia is OK: DISK OK - free space: /sql 132510 MB (11% inode=99%): [05:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:57:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128250 MB (10% inode=99%): [06:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:06:53] /sql on cassia is OK: DISK OK - free space: /sql 134483 MB (11% inode=99%): [06:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279974 MB (5% inode=55%): [06:17:15] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.49, 25.62, 25.84 [06:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 823112.000000 [06:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:46:13] Free Memory on damiana is WARNING: WARNING - 6.6% (548936 kB) free! [06:48:13] Free Memory on damiana is OK: OK - 7.1% (594372 kB) free. [06:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:03:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:13:13] Load avg. on nightshade is WARNING: WARNING - load average: 22.76, 21.90, 16.98 [07:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279612 MB (5% inode=55%): [07:18:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.25, 22.96, 24.05 [07:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 826713.000000 [07:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:20:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:20:11] Free Memory on damiana is WARNING: WARNING - 5.9% (492984 kB) free! [08:20:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:20:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:20:13] Free Memory on damiana is OK: OK - 7.3% (607556 kB) free. [08:20:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:20:14] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:20:14] Load avg. on nightshade is WARNING: WARNING - load average: 15.83, 18.03, 18.26 [08:20:15] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279281 MB (5% inode=55%): [08:20:17] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.91, 23.01, 23.71 [08:20:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 830313.000000 [08:26:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 121648 MB (10% inode=99%): [08:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:41:53] /sql on cassia is OK: DISK OK - free space: /sql 131673 MB (11% inode=99%): [08:49:52] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129340 MB (10% inode=99%): [08:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:06:13] Load avg. on nightshade is OK: OK - load average: 9.53, 12.17, 14.99 [09:07:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:08:53] /sql on cassia is OK: DISK OK - free space: /sql 133875 MB (11% inode=99%): [09:08:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:10:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:13:13] Load avg. on nightshade is WARNING: WARNING - load average: 22.75, 19.96, 17.61 [09:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 278648 MB (5% inode=55%): [09:15:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129980 MB (10% inode=99%): [09:16:53] /sql on cassia is OK: DISK OK - free space: /sql 130704 MB (11% inode=99%): [09:18:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.39, 24.79, 25.10 [09:18:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 833913.000000 [09:33:13] Free Memory on damiana is WARNING: WARNING - 6.1% (507056 kB) free! [09:36:14] Free Memory on damiana is OK: OK - 7.3% (615120 kB) free. [09:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:43:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 127755 MB (10% inode=99%): [09:47:53] /sql on cassia is OK: DISK OK - free space: /sql 130763 MB (11% inode=99%): [09:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:50:14] Free Memory on damiana is WARNING: WARNING - 5.9% (496168 kB) free! [09:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:57:13] Load avg. on nightshade is OK: OK - load average: 14.32, 14.15, 14.98 [10:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:07:13] Load avg. on nightshade is WARNING: WARNING - load average: 15.08, 14.94, 15.05 [10:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 278033 MB (5% inode=55%): [10:21:55] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.30, 25.54, 26.27 [10:21:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 837513.000000 [10:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:50:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:56:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129877 MB (10% inode=99%): [11:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:07:13] Load avg. on nightshade is WARNING: WARNING - load average: 16.47, 16.33, 16.03 [11:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 277326 MB (5% inode=55%): [11:18:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.21, 25.27, 25.32 [11:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 841112.000000 [11:19:14] Load avg. on nightshade is OK: OK - load average: 12.48, 13.83, 14.94 [11:22:13] Free Memory on damiana is WARNING: WARNING - 6.0% (500720 kB) free! [11:28:14] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (381960 kB) free! [11:30:14] Free Memory on damiana is WARNING: WARNING - 5.8% (483768 kB) free! [11:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:08:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 276582 MB (5% inode=55%): [12:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 844713.000000 [12:18:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.91, 28.38, 28.14 [12:20:53] hello all [12:21:57] hi dab [12:22:12] i just got an email saying my SGE job had an issue [12:22:49] http://dpaste.de/1A3i9/raw/ [12:25:14] Load avg. on nightshade is WARNING: WARNING - load average: 14.75, 17.62, 16.08 [12:26:45] duh: try to reach Merlissimo [12:28:35] ok [12:31:13] Free Memory on damiana is WARNING: WARNING - 6.2% (515396 kB) free! [12:32:14] Load avg. on nightshade is OK: OK - load average: 13.26, 14.18, 14.92 [12:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:45:13] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (405136 kB) free! [12:46:13] Free Memory on damiana is WARNING: WARNING - 5.1% (426068 kB) free! [12:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:55:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:15:26] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 275023 MB (5% inode=55%): [13:18:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 848313.000000 [13:18:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.98, 25.60, 26.19 [13:32:14] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (334912 kB) free! [13:36:14] Free Memory on damiana is WARNING: WARNING - 5.4% (453584 kB) free! [13:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:47:13] Free Memory on damiana is OK: OK - 7.2% (606792 kB) free. [13:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:10:05] Silke Meyer * [Toolserver-l] Hello! New TS coordinator @ WMDE [14:10:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:11:02] o: [14:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 274452 MB (5% inode=55%): [14:18:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.23, 25.60, 25.88 [14:18:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 851913.000000 [14:21:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 78567 MB (8% inode=99%): [14:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127190 MB (20% inode=99%): [14:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:05:13] Free Memory on damiana is WARNING: WARNING - 6.6% (553600 kB) free! [15:06:14] Free Memory on damiana is OK: OK - 7.1% (597500 kB) free. [15:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 273799 MB (5% inode=55%): [15:18:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.08, 26.29, 26.71 [15:18:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 855513.000000 [16:42:13] Free Memory on damiana is WARNING: WARNING - 5.7% (474828 kB) free! [16:42:14] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (411704 kB) free! [16:42:16] I just spent ages finding out why my .basrc isn't being executed... didn't have a .bash_profile as it turned out -.- [16:42:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [16:42:20] /sql on cassia is OK: DISK OK - free space: /sql 130435 MB (11% inode=99%): [16:42:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126922 MB (20% inode=99%): [16:42:23] Tim Landscheidt * Re: [Toolserver-l] Mail forwarding not working [16:42:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:42:25] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129861 MB (10% inode=99%): [16:42:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:42:26] Free Memory on damiana is WARNING: WARNING - 5.6% (467600 kB) free! [16:42:29] /sql on cassia is OK: DISK OK - free space: /sql 131045 MB (11% inode=99%): [16:42:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:50:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129784 MB (10% inode=99%): [16:50:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 272237 MB (5% inode=55%): [17:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 862713.000000 [17:18:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.84, 26.39, 26.31 [17:29:13] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (239248 kB) free! [17:36:13] Free Memory on damiana is WARNING: WARNING - 5.1% (430136 kB) free! [17:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126358 MB (20% inode=99%): [17:44:13] Free Memory on damiana is OK: OK - 7.1% (591884 kB) free. [17:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:03:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 271474 MB (5% inode=55%): [18:18:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.11, 26.22, 26.07 [18:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 866313.000000 [18:25:13] Free Memory on damiana is WARNING: WARNING - 6.1% (512412 kB) free! [18:30:14] Free Memory on damiana is OK: OK - 7.3% (614348 kB) free. [18:32:53] /sql on cassia is OK: DISK OK - free space: /sql 130140 MB (11% inode=99%): [18:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126031 MB (20% inode=99%): [18:39:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128988 MB (10% inode=99%): [18:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:52:53] /sql on cassia is OK: DISK OK - free space: /sql 130432 MB (11% inode=99%): [18:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:05:13] Free Memory on damiana is WARNING: WARNING - 6.3% (531360 kB) free! [19:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:07:13] Free Memory on damiana is OK: OK - 7.1% (590996 kB) free. [19:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 270743 MB (5% inode=55%): [19:18:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.55, 25.79, 25.84 [19:18:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 869913.000000 [19:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:37:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129398 MB (10% inode=99%): [19:38:14] Free Memory on damiana is WARNING: WARNING - 5.7% (479900 kB) free! [19:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125737 MB (20% inode=99%): [19:41:13] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (415236 kB) free! [19:44:13] Free Memory on damiana is WARNING: WARNING - 5.2% (433088 kB) free! [19:50:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:05:53] /sql on cassia is OK: DISK OK - free space: /sql 130235 MB (11% inode=99%): [20:07:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:12:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129959 MB (10% inode=99%): [20:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 269419 MB (5% inode=55%): [20:18:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.82, 24.36, 25.65 [20:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 873513.000000 [20:36:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125500 MB (20% inode=99%): [20:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 268792 MB (5% inode=55%): [21:18:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.16, 20.38, 20.50 [21:18:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 877113.000000 [21:25:13] Free Memory on damiana is WARNING: WARNING - 6.4% (537508 kB) free! [21:30:13] Free Memory on damiana is OK: OK - 7.6% (633880 kB) free. [21:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:38:13] Load avg. on willow is WARNING: WARNING - load average: 18.87, 19.57, 19.98 [21:39:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125200 MB (20% inode=99%): [21:41:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.89, 20.10, 20.11 [21:44:13] Load avg. on willow is WARNING: WARNING - load average: 19.00, 19.74, 19.97 [21:47:14] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (415456 kB) free! [21:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:51:14] Free Memory on damiana is WARNING: WARNING - 6.0% (506892 kB) free! [21:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:56:13] Free Memory on damiana is OK: OK - 7.3% (612704 kB) free. [22:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:11:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.58, 20.86, 20.47 [22:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 268136 MB (5% inode=55%): [22:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 880712.000000 [22:27:13] Free Memory on damiana is WARNING: WARNING - 6.2% (521184 kB) free! [22:28:14] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (389072 kB) free! [22:29:13] Load avg. on willow is WARNING: WARNING - load average: 18.02, 19.33, 19.93 [22:30:14] Free Memory on damiana is WARNING: WARNING - 5.6% (468180 kB) free! [22:30:25] hoo|away: Heh, yeah. Depending on the system, sometimes you need both. Even if one is just a symlink to the other. [22:30:42] I've seen strangeness where screen wouldn't respect my customizations in .bash_profile, for example. [22:31:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.95, 20.47, 20.27 [22:33:13] Load avg. on willow is WARNING: WARNING - load average: 18.12, 19.49, 19.93 [22:33:28] Susan: I've renamd it not to .bash_profile and it does what I want :) [22:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124923 MB (20% inode=99%): [22:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:01:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.95, 22.66, 20.75 [23:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:11:13] Free Memory on damiana is WARNING: WARNING - 6.6% (551772 kB) free! [23:15:14] Free Memory on damiana is OK: OK - 7.5% (624384 kB) free. [23:15:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 266566 MB (5% inode=55%): [23:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 884313.000000 [23:25:13] Load avg. on willow is WARNING: WARNING - load average: 18.77, 19.33, 19.93 [23:31:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.20, 21.84, 20.66 [23:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124541 MB (20% inode=99%): [23:40:13] Load avg. on willow is WARNING: WARNING - load average: 18.88, 19.53, 19.97 [23:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL