[00:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:01:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:03:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 99333 MB (16% inode=99%): [00:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [00:23:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 300726 MB (5% inode=30%): [00:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [00:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.89, 26.29, 26.24 [00:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:57:43] / on willow is WARNING: DISK WARNING - free space: / 19095 MB (18% inode=98%): [00:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 19095 MB (18% inode=98%): [01:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:02:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 99142 MB (16% inode=99%): [01:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [01:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:24:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 301386 MB (5% inode=30%): [01:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [01:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:50:06] DaB. * Re: [Toolserver-l] Downtime of s7 starting tomorrow [01:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.83, 27.36, 27.03 [01:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:57:43] / on willow is WARNING: DISK WARNING - free space: / 19045 MB (18% inode=98%): [01:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19045 MB (18% inode=98%): [01:58:06] DaB. * Re: [Toolserver-l] Downtime of s7 starting tomorrow [02:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:02:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97243 MB (15% inode=99%): [02:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [02:23:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:24:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294127 MB (5% inode=30%): [02:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [02:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:50:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 88073 MB (9% inode=99%): [02:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.60, 27.04, 26.76 [02:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:57:42] / on willow is WARNING: DISK WARNING - free space: / 19000 MB (18% inode=98%): [02:57:53] /tmp on willow is WARNING: DISK WARNING - free space: / 19000 MB (18% inode=98%): [03:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:02:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:03:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98956 MB (16% inode=99%): [03:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [03:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:24:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 297297 MB (5% inode=30%): [03:46:45] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [03:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.90, 29.66, 29.79 [03:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:57:43] / on willow is WARNING: DISK WARNING - free space: / 18934 MB (18% inode=98%): [03:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 18934 MB (18% inode=98%): [04:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:03:05] @replag [04:03:05] md_5: s1-rr-a-wd: 12s [-0.00 s/s]; s1-user-c: 33s [+0.00 s/s]; s1-user-wd: 12s [-0.00 s/s]; s2-user: 19s [+0.00 s/s]; s2-user-c: 35s [-0.01 s/s]; s2-user-wd: 12s [+0.00 s/s]; s3-user: 32s [+0.00 s/s]; s4-rr-a: 27s [+0.00 s/s] [04:03:06] md_5: s4-user: 27s [+0.00 s/s]; s4-user-wd: 12s [-]; s5-user-c: 35s [-0.01 s/s]; s5-user-wd: 12s [+0.00 s/s]; s6-user-wd: 12s [-0.00 s/s]; s7-user: error; s7-user-wd: error [04:03:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:03:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98901 MB (16% inode=99%): [04:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [04:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:24:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 297643 MB (5% inode=30%): [04:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [04:49:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.63, 31.77, 32.04 [04:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:57:42] / on willow is WARNING: DISK WARNING - free space: / 18871 MB (18% inode=98%): [04:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 18871 MB (18% inode=98%): [05:00:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [05:04:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98825 MB (16% inode=99%): [05:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [05:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:24:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 295990 MB (5% inode=30%): [05:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [05:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.07, 30.77, 31.39 [05:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:57:42] / on willow is WARNING: DISK WARNING - free space: / 18828 MB (17% inode=98%): [05:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 18828 MB (17% inode=98%): [06:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [06:04:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98727 MB (16% inode=99%): [06:21:53] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [06:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:24:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 297808 MB (5% inode=30%): [06:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:32:02] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [06:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.93, 29.34, 30.24 [06:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:57:42] / on willow is WARNING: DISK WARNING - free space: / 18780 MB (17% inode=98%): [06:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 18780 MB (17% inode=98%): [07:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:03:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [07:05:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98600 MB (16% inode=99%): [07:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [07:23:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:24:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 296328 MB (5% inode=30%): [07:31:21] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:32:02] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:46:45] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [07:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.36, 26.03, 27.57 [07:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:57:43] / on willow is WARNING: DISK WARNING - free space: / 18726 MB (17% inode=98%): [07:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 18726 MB (17% inode=98%): [08:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:03:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [08:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98471 MB (16% inode=99%): [08:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [08:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:24:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 295384 MB (5% inode=30%): [08:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:32:12] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [08:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.88, 28.67, 29.33 [08:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:57:42] / on willow is WARNING: DISK WARNING - free space: / 18677 MB (17% inode=98%): [08:57:58] /tmp on willow is WARNING: DISK WARNING - free space: / 18677 MB (17% inode=98%): [09:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:04:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [09:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98303 MB (16% inode=99%): [09:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [09:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:24:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293235 MB (5% inode=30%): [09:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:33:12] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [09:49:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.57, 26.94, 27.66 [09:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:57:52] / on willow is WARNING: DISK WARNING - free space: / 18489 MB (17% inode=98%): [09:57:52] /tmp on willow is WARNING: DISK WARNING - free space: / 18489 MB (17% inode=98%): [09:59:52] Free Memory on damiana is WARNING: WARNING - 6.3% (526920 kB) free! [10:00:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:02:52] Free Memory on damiana is OK: OK - 7.2% (599600 kB) free. [10:04:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [10:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 98106 MB (16% inode=99%): [10:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [10:24:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:24:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291389 MB (5% inode=29%): [10:27:42] /sql on cassia is WARNING: DISK WARNING - free space: /sql 89016 MB (7% inode=99%): [10:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:32:52] / on willow is OK: DISK OK - free space: / 24480 MB (23% inode=99%): [10:32:53] /tmp on willow is OK: DISK OK - free space: / 24480 MB (23% inode=99%): [10:33:13] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:46:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [10:49:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.19, 29.20, 29.61 [10:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:04:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [11:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97924 MB (16% inode=99%): [11:21:53] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [11:24:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293272 MB (5% inode=30%): [11:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:33:12] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:38:35] @replag [11:38:39] Merlissimo: s1-rr-a-wd: 21s [+0.00 s/s]; s1-user-wd: 22s [+0.00 s/s]; s2-user-wd: 24s [+0.00 s/s]; s3-user: 1m 17s [+0.00 s/s]; s4-user-wd: 25s [+0.00 s/s]; s5-user-wd: 25s [+0.00 s/s]; s6-user: 26s [-0.00 s/s]; s6-user-wd: 25s [+0.00 s/s] [11:38:40] Merlissimo: s7-user: error; s7-user-wd: error [11:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [11:49:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.61, 29.52, 30.08 [11:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:00:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [12:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:10:22] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [12:11:03] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [12:14:22] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2018 [12:15:22] MySQL slave on z-dat-s6-a is OK: Uptime: 641 Threads: 4 Questions: 3 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 8 Queries per second avg: 0.4 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:16:02] MySQL on z-dat-s6-a is OK: Uptime: 680 Threads: 5 Questions: 1435 Slow queries: 1 Opens: 79 Flush tables: 1 Open tables: 62 Queries per second avg: 2.110 [12:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97745 MB (16% inode=99%): [12:21:53] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [12:22:22] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2152 [12:25:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 297528 MB (5% inode=30%): [12:31:24] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:33:12] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:41:23] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3605 [12:44:23] MySQL slave on z-dat-s6-a is OK: Uptime: 2381 Threads: 12 Questions: 173175 Slow queries: 363 Opens: 339 Flush tables: 1 Open tables: 306 Queries per second avg: 72.732 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1658 [12:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [12:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.87, 32.53, 32.67 [12:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:04:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [13:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97523 MB (16% inode=99%): [13:21:53] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [13:25:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 298532 MB (5% inode=30%): [13:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:33:13] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:41:23] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5575 [13:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [13:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:52:22] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3468 [13:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.35, 31.79, 32.54 [13:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:01:22] MySQL slave on z-dat-s3-a is OK: Uptime: 1381784 Threads: 19 Questions: 1300482703 Slow queries: 44603 Opens: 7001700 Flush tables: 1 Open tables: 16384 Queries per second avg: 941.162 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1690 [14:04:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97315 MB (16% inode=99%): [14:21:52] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [14:25:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 299775 MB (5% inode=30%): [14:31:25] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:33:13] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:46:42] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [14:50:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 87895 MB (9% inode=99%): [14:50:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.82, 30.82, 31.74 [14:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:00:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:04:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:20:02] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:13] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:13] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:52] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97106 MB (15% inode=99%): [15:21:02] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:21:03] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:21:53] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [15:25:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 304399 MB (5% inode=30%): [15:28:17] s3 will be down for a short moment [15:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:31:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:33:12] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:46:12] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:46:38] emacs was updated – maybe it has finally an editor on board ;) [15:46:42] MySQL on z-dat-s7-a is CRITICAL: Access denied for user tsnagios7643@damiana-bge0.esi.toolserver.org (using password: YES) [15:46:53] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:47:22] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2070 [15:47:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2071.000000 [15:47:22] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:50:42] MySQL on z-dat-s7-a is OK: Uptime: 756 Threads: 4 Questions: 4091 Slow queries: 0 Opens: 145 Flush tables: 1 Open tables: 10 Queries per second avg: 5.411 [15:50:53] MySQL slave on z-dat-s7-a is WARNING: No slaves defined [15:50:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:54:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.39, 34.31, 34.27 [15:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:58:24] @replag [15:58:27] DaBPunkt: s1-rr-a-wd: 21m 3s [+0.08 s/s]; s1-user: 31m 21s [+0.01 s/s]; s1-user-wd: 18m 1s [+0.07 s/s]; s2-user-wd: 23m 18s [+0.09 s/s]; s3-user: 24s [-0.00 s/s]; s4-user-wd: 4m 22s [+0.02 s/s]; s5-user-wd: 23m 18s [+0.09 s/s]; s6-user-wd: 2h 57m 54s [+0.68 s/s] [15:58:28] DaBPunkt: s7-user: error; s7-user-wd: error [16:00:22] MySQL slave on rosemary is OK: Uptime: 2312346 Threads: 9 Questions: 1587345026 Slow queries: 481976 Opens: 149027 Flush tables: 1 Open tables: 3682 Queries per second avg: 686.465 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1742 [16:00:22] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1736.000000 [16:01:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:15:23] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [16:17:23] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2167.000000 [16:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96910 MB (15% inode=99%): [16:25:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 303843 MB (5% inode=30%): [16:41:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [16:50:53] MySQL slave on z-dat-s7-a is WARNING: No slaves defined [16:50:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.92, 34.30, 34.59 [16:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:55:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 226728 [16:57:34] @replag [16:57:34] DaBPunkt: s1-rr-a-wd: 1h 20m 12s [+1.00 s/s]; s1-user: 1h 16m 26s [+0.76 s/s]; s1-user-wd: 15s [-0.30 s/s]; s2-user: 10s [-0.00 s/s]; s2-user-c: 1m 9s [+0.00 s/s]; s2-user-wd: 14s [-0.39 s/s]; s3-user: 27s [+0.00 s/s]; s4-user-wd: 14s [-0.07 s/s] [16:57:35] DaBPunkt: s5-user-c: 1m 10s [+0.00 s/s]; s5-user-wd: 15s [-0.39 s/s]; s6-user-wd: 2h 31m 50s [-0.44 s/s]; s7-user: error; s7-user-wd: error [17:00:01] @replag [17:00:01] DaBPunkt: s1-rr-a-wd: 1h 22m 40s [+1.00 s/s]; s1-user: 1h 18m 54s [+1.01 s/s]; s1-user-wd: 2m 24s [+0.88 s/s]; s2-user: 20s [+0.07 s/s]; s2-user-wd: 2m 20s [+0.86 s/s]; s4-user-wd: 2m 24s [+0.88 s/s]; s5-user-wd: 2m 20s [+0.85 s/s]; s6-user-wd: 2h 34m 7s [+0.93 s/s] [17:00:03] DaBPunkt: s7-user: error; s7-user-wd: error [17:01:20] @replag [17:01:20] DaBPunkt: s1-rr-a-wd: 1h 23m 59s [+1.00 s/s]; s1-user: 1h 20m 13s [+1.00 s/s]; s1-user-wd: 3m 43s [+1.00 s/s]; s2-user-wd: 3m 39s [+1.00 s/s]; s3-user: 20s [-0.03 s/s]; s4-user-wd: 3m 43s [+1.00 s/s]; s5-user-wd: 3m 39s [+1.00 s/s]; s6-user-wd: 2h 35m 26s [+1.00 s/s] [17:01:22] DaBPunkt: s7-user: error; s7-user-wd: error [17:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:06:38] ok, we are done :) [17:15:22] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [17:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:21:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96685 MB (15% inode=99%): [17:25:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 304171 MB (5% inode=30%): [17:41:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7207.000000 [17:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.23, 36.45, 36.36 [17:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:55:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 193798 [18:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:15:24] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [18:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96547 MB (15% inode=99%): [18:25:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 307302 MB (5% inode=31%): [18:41:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10807.000000 [18:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.80, 36.45, 36.46 [18:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:55:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 165598 [19:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:15:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11440 [19:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96342 MB (15% inode=99%): [19:25:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 306620 MB (5% inode=31%): [19:41:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10662.000000 [19:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.87, 35.68, 35.47 [19:54:52] Free Memory on damiana is WARNING: WARNING - 6.9% (580144 kB) free! [19:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:55:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136052 [19:55:52] Free Memory on damiana is OK: OK - 7.1% (592712 kB) free. [20:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:15:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11023 [20:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96046 MB (15% inode=99%): [20:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 305783 MB (5% inode=30%): [20:41:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9990.000000 [20:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.46, 34.59, 35.26 [20:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:55:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 100866 [21:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:15:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8819 [21:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95846 MB (15% inode=99%): [21:21:11] @replag [21:21:11] Merlissimo: s1-rr-a-wd: 1m 37s [-0.32 s/s]; s1-user: 2h 24m 23s [+0.25 s/s]; s1-user-c: 1m 44s [+0.00 s/s]; s1-user-wd: 1m 37s [-0.01 s/s]; s2-user-c: 4m 50s [+0.01 s/s]; s2-user-wd: 1m 52s [-0.01 s/s]; s3-user: 14s [-0.00 s/s]; s4-user-wd: 1m 37s [-0.01 s/s] [21:21:12] Merlissimo: s5-user-c: 4m 50s [+0.01 s/s]; s5-user-wd: 1m 52s [-0.01 s/s]; s6-user-wd: 1m 37s [-0.59 s/s]; s7-user: 1d 1h 48m 51s [+0.17 s/s]; s7-user-wd: 1m 37s [+0.00 s/s] [21:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 305610 MB (5% inode=30%): [21:41:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8115.000000 [21:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.51, 36.82, 36.73 [21:54:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:55:53] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78532 [22:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:15:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6178 [22:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95636 MB (15% inode=99%): [22:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 305716 MB (5% inode=30%): [22:27:48] /sql on cassia is WARNING: DISK WARNING - free space: /sql 91881 MB (7% inode=99%): [22:41:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4667.000000 [22:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 38.91, 37.89, 37.31 [22:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:55:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45124 [23:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [23:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:07:38] Hello all! I can't create a JIRA account, it gives a "System error occured while trying to create an account" error and I need to make one to apply for a Toolserver account. [23:07:47] Is this a known issue? [23:10:22] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3565 [23:10:23] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3556.000000 [23:11:36] Nevermind - working now [23:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95487 MB (15% inode=99%): [23:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 305862 MB (5% inode=30%): [23:48:22] MySQL slave on rosemary is OK: Uptime: 2340426 Threads: 33 Questions: 1615151892 Slow queries: 488853 Opens: 150593 Flush tables: 1 Open tables: 3693 Queries per second avg: 690.110 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1737 [23:48:23] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1738.000000 [23:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.79, 35.16, 35.66 [23:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:55:52] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25947