[00:13:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7213.000000 [00:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 403104 MB (7% inode=69%): [00:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:23:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54785 MB (8% inode=99%): [00:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:27:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:08:13] nacht ts [01:08:56] @replag [01:08:56] DaBPunkt: s1-rr-a-wd: 52s [+0.00 s/s]; s1-user-wd: 53s [+0.00 s/s]; s2-rr: 9m 34s [+0.04 s/s]; s2-user: 9m 34s [+0.04 s/s]; s2-user-c: error; s2-user-wd: 2h 55m 39s [+0.67 s/s]; s3-user: 1m 11s [+0.00 s/s]; s4-user-wd: 50s [+0.00 s/s] [01:08:57] DaBPunkt: s5-user-c: 2m 19s [-0.14 s/s]; s5-user-wd: 50s [+0.00 s/s]; s6-user-wd: 53s [+0.00 s/s]; s7-user-wd: 52s [+0.00 s/s] [01:13:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10814.000000 [01:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 406343 MB (7% inode=69%): [01:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:23:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54667 MB (8% inode=99%): [01:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:40:15] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2033 [01:45:15] MySQL slave on z-dat-s2-b is OK: Uptime: 109109 Threads: 9 Questions: 77336493 Slow queries: 1357 Opens: 3441410 Flush tables: 1 Open tables: 64 Queries per second avg: 708.800 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1681 [01:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:13:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12303.000000 [02:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 405761 MB (7% inode=69%): [02:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:56] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:23:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:26:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54797 MB (8% inode=99%): [02:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:47:35] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3560.000000 [02:53:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 404478 MB (7% inode=69%): [03:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:22:55] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:55] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:23:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:23:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:26:35] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1736.000000 [03:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54579 MB (8% inode=99%): [03:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:05:14] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70013 MB (7% inode=99%): [04:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 405011 MB (7% inode=69%): [04:22:34] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:22:56] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:56] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:23:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54522 MB (8% inode=99%): [04:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:18:37] [[Special:Log/newusers]] create 10 * Bisawajit * (New user account) [05:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 404232 MB (7% inode=69%): [05:22:34] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54469 MB (8% inode=99%): [05:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 404111 MB (7% inode=69%): [06:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:06] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:23:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:23:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54379 MB (8% inode=99%): [06:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:27:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 403913 MB (7% inode=69%): [07:22:36] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:24:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54292 MB (8% inode=99%): [07:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:17:45] [[Special:Log/newusers]] create 10 * Timeintervalanalyzer * (New user account) [08:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 407700 MB (7% inode=69%): [08:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:24:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54125 MB (8% inode=99%): [08:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:27:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:53:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 406475 MB (7% inode=69%): [09:22:35] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:24:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53867 MB (8% inode=99%): [09:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:54:35] /sql on cassia is WARNING: DISK WARNING - free space: /sql 92955 MB (7% inode=99%): [10:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 405376 MB (7% inode=69%): [10:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:23:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:24:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53621 MB (8% inode=99%): [10:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 404194 MB (7% inode=69%): [11:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:24:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53379 MB (8% inode=99%): [11:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:08:15] Load avg. on ortelius is WARNING: WARNING - load average: 12.53, 18.90, 12.61 [12:10:15] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 33.90, 22.53, 14.62 [12:12:14] Load avg. on ortelius is WARNING: WARNING - load average: 17.91, 21.88, 15.49 [12:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 403073 MB (7% inode=69%): [12:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53134 MB (8% inode=99%): [12:27:15] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 35.99, 20.77, 17.40 [12:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:27:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:39:15] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2015 [12:40:25] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 56300 MB (5% inode=99%): [12:43:15] /sql on thyme is WARNING: DISK WARNING - free space: /sql 91350 MB (9% inode=99%): [12:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 401835 MB (7% inode=69%): [13:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:24:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52858 MB (8% inode=99%): [13:27:15] MySQL slave on z-dat-s2-b is OK: Uptime: 151229 Threads: 8 Questions: 111737279 Slow queries: 1925 Opens: 4641268 Flush tables: 1 Open tables: 64 Queries per second avg: 738.861 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1669 [13:27:16] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 73.43, 58.83, 38.26 [13:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:38:25] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 72503 MB (7% inode=99%): [13:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:58:15] /sql on thyme is OK: DISK OK - free space: /sql 129908 MB (13% inode=99%): [14:17:15] Load avg. on ortelius is WARNING: WARNING - load average: 14.21, 12.38, 19.80 [14:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 400722 MB (7% inode=69%): [14:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52570 MB (8% inode=99%): [14:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:31:15] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 46.25, 22.97, 19.46 [14:33:15] Load avg. on ortelius is WARNING: WARNING - load average: 16.52, 20.38, 19.00 [14:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 399015 MB (7% inode=69%): [15:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:06] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [15:23:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52265 MB (8% inode=99%): [15:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:38:15] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 33.59, 40.29, 31.50 [15:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 396017 MB (7% inode=69%): [16:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:06] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:06] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:24:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:25:14] Load avg. on ortelius is WARNING: WARNING - load average: 11.20, 14.38, 19.47 [16:26:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51990 MB (8% inode=99%): [16:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:55:15] Load avg. on ortelius is OK: OK - load average: 9.84, 12.96, 14.78 [17:06:07] Hi, would someone mind checking some diffs for me from Meta-Wiki? [17:06:16] Just to see if there's anything abnormal about them [17:06:43] The diffs here: https://bugzilla.wikimedia.org/show_bug.cgi?id=45081 [17:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 394724 MB (7% inode=69%): [17:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51675 MB (8% inode=99%): [17:27:14] Load avg. on ortelius is WARNING: WARNING - load average: 15.62, 18.30, 15.46 [17:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:32:14] Load avg. on ortelius is OK: OK - load average: 7.39, 13.86, 14.61 [17:37:06] Sumana Harihareswara * [Toolserver-l] Coren joining WMF Ops department as Tools Labs contractor [17:39:30] Ah, didn't see that one yet. [17:40:10] * Coren waves. [17:42:26] Coren: Toutes mes félicitations! :-) [17:42:57] scfc_de: Merci. [17:49:36] Coren: Eh bien bravo aussi :-) D'avance merci pour ton aide héhé [17:52:50] Darkdadaah: Il y a beacoup a faire, mais je crois bien que l'enjeu en vaut la peine. [17:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:53:15] Load avg. on ortelius is WARNING: WARNING - load average: 17.33, 16.94, 15.91 [17:59:15] Load avg. on ortelius is OK: OK - load average: 10.84, 13.98, 14.99 [18:00:53] Coren: By the way, can you add me to the webtools project? [18:01:32] Darkdadaah: I have the ability to do so, I don't know whether I have the /authority/ to do so just yet. Lemme look into it. [18:03:28] Darkdadaah: Do you already have a labsconsole account? [18:03:49] Yes, Darkdadaah as well [18:05:24] Darkdadaah: Added. [18:06:08] Thanks! [18:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 393362 MB (7% inode=69%): [18:22:45] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:04] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:06] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:25:15] Load avg. on ortelius is WARNING: WARNING - load average: 19.79, 16.38, 16.48 [18:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51379 MB (8% inode=99%): [18:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:31:15] Load avg. on ortelius is OK: OK - load average: 6.75, 11.89, 14.54 [18:32:01] Coren: So starting on the 25 we'll be able to bug you with our nooby requests? [18:33:16] Darkdadaah: I expect I shall be busy in preparing infrastructure for a couple of weeks, but yeah, that's what I'll be there for. :-) [18:37:02] I won't bother you just yet, it looks like I will have my hands full with scribunto which should come next week. [18:39:15] Load avg. on ortelius is WARNING: WARNING - load average: 15.46, 15.47, 16.18 [18:42:14] Load avg. on ortelius is OK: OK - load average: 6.48, 12.23, 14.83 [18:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 392486 MB (7% inode=69%): [19:22:44] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:05] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:05] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51045 MB (8% inode=99%): [19:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:27:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:28:55] SMTP on ortelius is OK: SMTP OK - 0.010 sec. response time [19:28:55] SMTP on wolfsbane is OK: SMTP OK - 0.120 sec. response time [19:30:35] SMTP on willow is OK: SMTP OK - 0.056 sec. response time [19:53:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:01:15] Load avg. on ortelius is WARNING: WARNING - load average: 17.79, 23.99, 17.91 [20:05:15] Load avg. on ortelius is OK: OK - load average: 6.09, 13.54, 14.82 [20:21:15] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 73.45, 44.02, 26.73 [20:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 391275 MB (7% inode=69%): [20:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50691 MB (8% inode=99%): [20:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:53:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:21:14] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 12.14, 18.50, 24.18 [21:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 390143 MB (7% inode=69%): [21:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:23:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:24:49] seems everything is going down into small pieces here ヾ [21:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50368 MB (8% inode=99%): [21:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:54:35] /sql on cassia is WARNING: DISK WARNING - free space: /sql 93378 MB (7% inode=99%): [21:56:43] AzaToth: I think most users just put tsnag on ignore :-). [22:01:15] Load avg. on ortelius is WARNING: WARNING - load average: 14.18, 14.42, 19.85 [22:06:10] hehe [22:10:14] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 27.61, 20.34, 20.12 [22:18:15] Load avg. on ortelius is WARNING: WARNING - load average: 14.68, 18.62, 19.95 [22:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 388749 MB (7% inode=69%): [22:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:23:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:23:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:23:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:24:06] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:24:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50092 MB (8% inode=99%): [22:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:37:51] I really don't blame them [22:38:15] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 7.29, 18.25, 24.06 [22:51:26] [[Special:Log/newusers]] create 10 * 三毛猫 * (New user account) [23:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 385447 MB (7% inode=69%): [23:23:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:23:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:23:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:24:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:24:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:24:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:26:56] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49819 MB (8% inode=99%): [23:27:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:38:15] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 23.43, 24.00, 27.56