[00:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 379501 MB (7% inode=66%): [00:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:22:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:35] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33458.000000 [00:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:57] nacht ts [00:25:37] @replag [00:25:37] DaBPunkt: s1-rr-a-wd: 56s [-0.00 s/s]; s1-user-wd: 55s [-0.00 s/s]; s2-rr: 1m 14s [-0.00 s/s]; s2-user: 1m 14s [-0.00 s/s]; s2-user-c: error; s2-user-wd: 1m 53s [-0.60 s/s]; s3-user: 22s [+0.00 s/s]; s4-user-wd: 57s [-0.00 s/s] [00:25:38] DaBPunkt: s5-user-c: 9h 13m 42s [-2.39 s/s]; s5-user-wd: 60s [-0.00 s/s]; s6-user-wd: 58s [-0.00 s/s]; s7-user-wd: 55s [-0.00 s/s] [00:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60109 MB (9% inode=99%): [00:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:27:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:30:18] did I miss the story? [00:30:27] what happened Krinkle? [00:30:39] dschwen: what story? [00:30:47] IBrokeWikipedia [00:30:56] dschwen: no, no story. [00:31:00] I just reserved the nick name. [00:31:21] you are a nick squatter! [00:31:37] are you going to sell it to the highest bidder? [00:32:34] / on willow is WARNING: DISK WARNING - free space: / 21661 MB (20% inode=98%): [00:32:35] /tmp on willow is WARNING: DISK WARNING - free space: / 21661 MB (20% inode=98%): [01:20:02] Krinkle: That nick should belong to someone in Ops. :-) [01:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 385034 MB (7% inode=66%): [01:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:22:35] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29046.000000 [01:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59933 MB (9% inode=99%): [01:27:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:27:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:32:35] / on willow is WARNING: DISK WARNING - free space: / 21357 MB (20% inode=98%): [01:32:36] /tmp on willow is WARNING: DISK WARNING - free space: / 21357 MB (20% inode=98%): [02:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 387300 MB (7% inode=66%): [02:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:22:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:35] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24260.000000 [02:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59763 MB (9% inode=99%): [02:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:27:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:32:34] /tmp on willow is WARNING: DISK WARNING - free space: / 21058 MB (20% inode=98%): [02:32:36] / on willow is WARNING: DISK WARNING - free space: / 21058 MB (20% inode=98%): [03:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 389009 MB (7% inode=66%): [03:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:22:45] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:45] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6873.000000 [03:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:23:06] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:23:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:23:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59625 MB (9% inode=99%): [03:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:27:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:28:44] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2875.000000 [03:31:45] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1553.000000 [03:32:34] /tmp on willow is WARNING: DISK WARNING - free space: / 20781 MB (19% inode=98%): [03:32:35] / on willow is WARNING: DISK WARNING - free space: / 20781 MB (19% inode=98%): [04:05:15] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70243 MB (7% inode=99%): [04:07:25] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2001.000000 [04:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 392094 MB (7% inode=66%): [04:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:35] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:22:45] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:45] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:23:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:24:25] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [04:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59539 MB (9% inode=99%): [04:27:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:27:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:32:35] / on willow is WARNING: DISK WARNING - free space: / 20505 MB (19% inode=98%): [04:32:36] /tmp on willow is WARNING: DISK WARNING - free space: / 20505 MB (19% inode=98%): [05:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 394347 MB (7% inode=67%): [05:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:22:35] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:26:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59474 MB (9% inode=99%): [05:27:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:32:35] / on willow is WARNING: DISK WARNING - free space: / 20239 MB (19% inode=98%): [05:33:35] /tmp on willow is WARNING: DISK WARNING - free space: / 20233 MB (19% inode=98%): [05:53:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 396909 MB (7% inode=67%): [06:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:22:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59388 MB (9% inode=99%): [06:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:32:35] / on willow is WARNING: DISK WARNING - free space: / 19960 MB (19% inode=98%): [06:33:35] /tmp on willow is WARNING: DISK WARNING - free space: / 19955 MB (19% inode=98%): [06:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 399397 MB (7% inode=67%): [07:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:56] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:26:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59273 MB (9% inode=99%): [07:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:32:35] / on willow is WARNING: DISK WARNING - free space: / 19676 MB (18% inode=98%): [07:33:35] /tmp on willow is WARNING: DISK WARNING - free space: / 19672 MB (18% inode=98%): [07:41:35] / on willow is OK: DISK OK - free space: / 22788 MB (21% inode=99%): [07:41:35] /tmp on willow is OK: DISK OK - free space: / 22788 MB (21% inode=99%): [07:53:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 401556 MB (7% inode=67%): [08:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:56] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:23:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:23:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59078 MB (9% inode=99%): [08:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 403193 MB (7% inode=67%): [09:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:23:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58788 MB (9% inode=99%): [09:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:54:25] /sql on cassia is WARNING: DISK WARNING - free space: /sql 97481 MB (8% inode=99%): [09:56:06] Krinkle * [Toolserver-l] Cron errors at hawthorn for qcronsub [10:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 404278 MB (7% inode=67%): [10:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:23:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58486 MB (9% inode=99%): [10:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:56:14] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2115 [11:09:15] MySQL slave on z-dat-s2-b is OK: Uptime: 56548 Threads: 11 Questions: 40281248 Slow queries: 680 Opens: 1588906 Flush tables: 1 Open tables: 64 Queries per second avg: 712.337 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1713 [11:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 406331 MB (7% inode=68%): [11:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:22:56] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:56] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58228 MB (9% inode=99%): [11:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 408229 MB (7% inode=68%): [12:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:23:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57958 MB (9% inode=99%): [12:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:48:06] DaB. * Re: [Toolserver-l] Cron errors at hawthorn for qcronsub [12:48:43] hello all [12:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 410189 MB (7% inode=68%): [13:22:34] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57709 MB (9% inode=99%): [13:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:27:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:54:11] @replag [13:54:12] liangent: s1-rr-a-wd: 1m 8s [+0.00 s/s]; s1-user-wd: 1m 8s [+0.00 s/s]; s2-rr: 16m 37s [+0.02 s/s]; s2-user: 16m 37s [+0.02 s/s]; s2-user-c: error; s2-user-wd: 1m 9s [-0.00 s/s]; s3-user: 49s [+0.00 s/s]; s4-user-wd: 1m 8s [+0.00 s/s] [13:54:13] liangent: s5-user-c: 1m 15s [-0.68 s/s]; s5-user-wd: 1m 2s [+0.00 s/s]; s6-user-wd: 1m 2s [+0.00 s/s]; s7-user-wd: 1m 8s [+0.00 s/s] [14:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 410846 MB (7% inode=68%): [14:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:23:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:26:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57453 MB (9% inode=99%): [14:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:53:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 410309 MB (7% inode=68%): [15:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:22:54] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57175 MB (9% inode=99%): [15:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:05:15] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70245 MB (7% inode=99%): [16:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 409704 MB (7% inode=68%): [16:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56902 MB (9% inode=99%): [16:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:39:15] Load avg. on ortelius is WARNING: WARNING - load average: 6.52, 16.26, 12.97 [16:40:15] Load avg. on ortelius is OK: OK - load average: 4.50, 13.92, 12.35 [16:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 408968 MB (7% inode=68%): [17:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56601 MB (9% inode=99%): [17:27:15] DaBPunkt: hi! are you around ? [17:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:28:10] I have a problem in trying connecting to willow… :/ [17:28:31] I cannot manage connecting to willow, yet I can connect to nightshade :| [17:32:43] Toto_Azero: connecting to willow is really slow lately, but I end up connected after a while. [17:33:32] Darkdadaah: doh yes right ! I've finally managed to get connected ^^ [17:34:42] Alright ^^ Though it would be good to know why it takes so long. [17:38:49] yes, I still wonder… [17:49:45] 28815 invadina 1 50 2 20M 18M cpu/3 108:02 12.57% python [17:49:45] 6454 hercule 1 1 17 673M 668M cpu/5 674:38 10.30% python [17:49:46] wow ! [17:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 408614 MB (7% inode=68%): [18:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54504 MB (8% inode=99%): [18:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:27:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:39:06] Maciej Jaros * Re: [Toolserver-l] Short downtime for s2 today [18:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 408296 MB (7% inode=68%): [19:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:56] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:23:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:24:47] DaBPunkt: any news on amaranth? [19:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56065 MB (9% inode=99%): [19:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 407844 MB (7% inode=68%): [20:22:34] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:22:54] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:25:45] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2072.000000 [20:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55803 MB (9% inode=99%): [20:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:39:06] DaB. * Re: [Toolserver-l] Short downtime for s2 today [20:48:38] HenriqueCrang: repara.. [20:48:42] @replag [20:48:43] Alchimista: s1-rr-a: 10s [-0.00 s/s]; s1-rr-a-wd: 46s [-0.00 s/s]; s1-user: 26s [-0.00 s/s]; s1-user-wd: 45s [-0.00 s/s]; s2-rr: 19s [-0.04 s/s]; s2-user: 19s [-0.04 s/s]; s2-user-c: error; s2-user-wd: 1m 24s [+0.00 s/s] [20:48:44] Alchimista: s4-user-wd: 46s [-0.00 s/s]; s5-user-c: 38m 17s [+0.09 s/s]; s5-user-wd: 46s [-0.00 s/s]; s6-user-wd: 44s [-0.00 s/s]; s7-user-wd: 46s [-0.00 s/s] [20:48:56] good [20:49:17] a nossa pedia está em s2-rr se n estou em erro [20:49:52] lembrarei disso [20:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:54:06] Platonides * Re: [Toolserver-l] Short downtime for s2 today [21:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 407027 MB (7% inode=68%): [21:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:23:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:23:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:25:45] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3197.000000 [21:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55457 MB (9% inode=99%): [21:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:54:35] /sql on cassia is WARNING: DISK WARNING - free space: /sql 94486 MB (7% inode=99%): [21:56:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3613.000000 [21:58:45] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3561.000000 [22:10:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3603.000000 [22:22:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 406467 MB (7% inode=68%): [22:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:26:56] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55201 MB (9% inode=99%): [22:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:38:45] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3555.000000 [22:48:45] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1759.000000 [22:49:35] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2173.000000 [22:53:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:13:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3613.000000 [23:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 405365 MB (7% inode=69%): [23:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:23:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54961 MB (8% inode=99%): [23:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:53:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL