[00:04:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:04:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 72332 MB (1% inode=68%): [00:04:49] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8859.000000 [00:05:08] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10371.000000 [00:05:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:05:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:05:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113423 MB (18% inode=99%): [00:05:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:05:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:05:41] aliasd on mayapple is CRITICAL: Connection refused [00:05:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:07:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:46:35] nacht ts [00:47:29] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1848 [00:49:29] MySQL slave on z-dat-s2-b is OK: Uptime: 431603 Threads: 10 Questions: 668680155 Slow queries: 6591 Opens: 3968734 Flush tables: 1 Open tables: 256 Queries per second avg: 1549.294 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1747 [01:00:29] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2158 [01:04:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11577.000000 [01:05:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:05:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:05:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113200 MB (18% inode=99%): [01:05:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 72897 MB (1% inode=68%): [01:05:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11246.000000 [01:06:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:06:42] aliasd on mayapple is CRITICAL: Connection refused [01:06:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:08:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:33:30] MySQL slave on z-dat-s2-b is OK: Uptime: 434244 Threads: 11 Questions: 671489634 Slow queries: 6623 Opens: 3980112 Flush tables: 1 Open tables: 256 Queries per second avg: 1546.341 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1766 [01:35:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:44:41] Load avg. on ortelius is WARNING: WARNING - load average: 16.72, 15.37, 11.34 [01:46:41] Load avg. on ortelius is OK: OK - load average: 13.94, 14.79, 11.62 [02:04:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:05:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14375.000000 [02:05:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:05:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:05:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111146 MB (18% inode=99%): [02:05:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 72800 MB (1% inode=68%): [02:05:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10798.000000 [02:06:44] aliasd on mayapple is CRITICAL: Connection refused [02:06:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:06:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:08:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:16:40] Load avg. on ortelius is WARNING: WARNING - load average: 28.67, 23.57, 16.89 [02:28:50] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3528.000000 [02:33:50] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1463.000000 [02:35:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:35:40] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 26.67, 24.27, 20.36 [03:01:41] Load avg. on ortelius is WARNING: WARNING - load average: 3.65, 14.09, 19.96 [03:04:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:04:49] Free Memory on turnera is WARNING: WARNING - 5.7% (475968 kB) free! [03:05:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16286.000000 [03:05:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:05:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:05:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112831 MB (18% inode=99%): [03:06:40] aliasd on mayapple is CRITICAL: Connection refused [03:06:40] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:06:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:06:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 72005 MB (1% inode=68%): [03:07:40] Load avg. on ortelius is OK: OK - load average: 3.51, 6.84, 14.61 [03:08:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:21:50] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (378440 kB) free! [03:33:00] Free Memory on turnera is OK: OK - 88.4% (7410024 kB) free. [03:36:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:38:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57987 MB (5% inode=99%): [03:46:09] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 61377 MB (6% inode=99%): [04:04:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:05:28] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18954.000000 [04:05:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:05:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:05:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112733 MB (18% inode=99%): [04:06:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:07:00] aliasd on mayapple is CRITICAL: Connection refused [04:07:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:07:10] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 71937 MB (1% inode=68%): [04:09:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:36:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:04:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:05:29] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19767.000000 [05:05:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:05:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:05:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112667 MB (18% inode=99%): [05:07:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:07:00] aliasd on mayapple is CRITICAL: Connection refused [05:07:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:07:09] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 71667 MB (1% inode=68%): [05:09:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:36:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:04:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:05:29] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20964.000000 [06:05:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:05:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:05:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112569 MB (18% inode=99%): [06:06:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:06:59] aliasd on mayapple is CRITICAL: Connection refused [06:06:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:07:09] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 71645 MB (1% inode=68%): [06:09:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:36:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:01:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58115 MB (5% inode=99%): [07:02:09] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 59113 MB (6% inode=99%): [07:04:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:05:29] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22545.000000 [07:05:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:05:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:05:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112468 MB (18% inode=99%): [07:06:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:07:00] aliasd on mayapple is CRITICAL: Connection refused [07:07:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:07:10] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 71485 MB (1% inode=68%): [07:09:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:36:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:49:29] Load avg. on nightshade is WARNING: WARNING - load average: 15.61, 14.88, 11.19 [08:05:00] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:05:29] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24725.000000 [08:05:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:05:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:05:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112315 MB (18% inode=99%): [08:06:30] Load avg. on nightshade is OK: OK - load average: 7.16, 14.27, 14.23 [08:06:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:59] aliasd on mayapple is CRITICAL: Connection refused [08:06:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:07:09] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 71267 MB (1% inode=68%): [08:09:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:23:29] Load avg. on nightshade is WARNING: WARNING - load average: 14.76, 15.22, 13.86 [08:36:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:37:29] Load avg. on nightshade is OK: OK - load average: 9.93, 13.99, 14.89 [08:50:19] Free Memory on damiana is CRITICAL: CRITICAL - 3.4% (288456 kB) free! [08:57:54] Free Memory on damiana is OK: OK - 91.9% (7703532 kB) free. [09:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26373.000000 [09:05:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:05:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:05:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112165 MB (18% inode=99%): [09:07:03] aliasd on mayapple is CRITICAL: Connection refused [09:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:07:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 68825 MB (1% inode=68%): [09:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:11:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57347 MB (5% inode=99%): [09:14:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58747 MB (6% inode=99%): [09:47:11] [[Special:Log/newusers]] create 10 * Eliade * (New user account) [10:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24603.000000 [10:05:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:05:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:05:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112014 MB (18% inode=99%): [10:07:03] aliasd on mayapple is CRITICAL: Connection refused [10:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:07:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 70686 MB (1% inode=67%): [10:08:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57708 MB (5% inode=99%): [10:08:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:12:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 59573 MB (6% inode=99%): [10:35:53] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2197 [10:36:43] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [10:43:53] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2674.000000 [10:59:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [10:59:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3621 [11:05:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24271.000000 [11:05:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:05:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:05:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111874 MB (18% inode=99%): [11:07:03] aliasd on mayapple is CRITICAL: Connection refused [11:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 72680 MB (1% inode=67%): [11:08:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:59:54] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6874.000000 [12:00:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6931 [12:05:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26520.000000 [12:05:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:05:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:05:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111832 MB (18% inode=99%): [12:07:04] aliasd on mayapple is CRITICAL: Connection refused [12:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 87966 MB (1% inode=67%): [12:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:09:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:23:53] Load avg. on ptolemy is WARNING: WARNING - load average: 24.62, 20.78, 13.68 [12:34:54] Load avg. on ptolemy is OK: OK - load average: 8.61, 14.48, 14.81 [13:00:03] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [13:00:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10344 [13:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27433.000000 [13:05:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:05:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:05:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111795 MB (18% inode=99%): [13:06:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:07:03] aliasd on mayapple is CRITICAL: Connection refused [13:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 170588 MB (3% inode=67%): [13:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:09:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:00:33] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10553.000000 [14:00:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10531 [14:05:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29474.000000 [14:05:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:05:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:05:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111711 MB (18% inode=99%): [14:06:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:07:03] aliasd on mayapple is CRITICAL: Connection refused [14:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 170105 MB (3% inode=67%): [14:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:09:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:00:33] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3975.000000 [15:00:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3912 [15:03:54] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3529 [15:04:32] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3413.000000 [15:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31166.000000 [15:06:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:06:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:06:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111650 MB (18% inode=99%): [15:07:03] aliasd on mayapple is CRITICAL: Connection refused [15:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:07:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 169481 MB (3% inode=67%): [15:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:09:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:12:33] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1650.000000 [15:12:53] MySQL slave on z-dat-s5-b is OK: Uptime: 172515 Threads: 5 Questions: 33850675 Slow queries: 2338 Opens: 81792 Flush tables: 1 Open tables: 256 Queries per second avg: 196.218 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1622 [15:43:12] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2057.000000 [15:51:54] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2062 [16:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33130.000000 [16:06:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:06:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:06:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111581 MB (18% inode=99%): [16:07:04] aliasd on mayapple is CRITICAL: Connection refused [16:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 168265 MB (3% inode=67%): [16:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:09:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:18:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [16:22:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3623 [16:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:45:15] @replag [16:45:17] liangent: s1-rr-a-wd: 51s [-]; s1-user: 19s [-]; s1-user-wd: 51s [-]; s2-rr: 1h 21m 44s [-]; s2-user: 1h 21m 44s [-]; s2-user-c: error; s2-user-wd: 1h 24m 39s [-]; s3-user-wd: 44s [-] [16:45:18] liangent: s4-user-wd: 9h 39m 23s [-]; s5-user-c: error; s6-user-wd: 48s [-]; s7-user-wd: 42s [-] [17:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35813.000000 [17:06:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:06:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:06:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111384 MB (18% inode=99%): [17:07:03] aliasd on mayapple is CRITICAL: Connection refused [17:07:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [17:08:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 167715 MB (3% inode=67%): [17:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:09:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:18:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6869.000000 [17:22:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7008 [17:58:06] Maarten Dammers * [Toolserver-l] Wikimedia Hackathon Amsterdam 2013: Registration opened [18:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35321.000000 [18:06:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:06:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:06:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111099 MB (18% inode=99%): [18:06:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:07:03] aliasd on mayapple is CRITICAL: Connection refused [18:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:08:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 166937 MB (3% inode=67%): [18:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:09:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:18:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9955.000000 [18:22:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10141 [19:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37172.000000 [19:06:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:07:03] aliasd on mayapple is CRITICAL: Connection refused [19:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:07:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:07:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110854 MB (18% inode=99%): [19:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 166001 MB (3% inode=67%): [19:09:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:09:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:18:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12510.000000 [19:22:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12688 [19:37:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38662.000000 [20:06:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:07:03] aliasd on mayapple is CRITICAL: Connection refused [20:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:07:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:07:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110629 MB (18% inode=99%): [20:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 165646 MB (3% inode=67%): [20:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:09:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:18:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15430.000000 [20:22:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15668 [21:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40277.000000 [21:06:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:06:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:07:03] aliasd on mayapple is CRITICAL: Connection refused [21:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 165186 MB (3% inode=67%): [21:08:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:08:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110394 MB (18% inode=99%): [21:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:10:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:18:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16727.000000 [21:23:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13955 [22:05:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41712.000000 [22:06:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:06:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:07:03] aliasd on mayapple is CRITICAL: Connection refused [22:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 164952 MB (3% inode=67%): [22:08:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:08:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110141 MB (18% inode=99%): [22:09:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:10:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:12:12] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 59494 MB (6% inode=99%): [22:18:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17245.000000 [22:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12787 [22:48:03] Load avg. on ortelius is WARNING: WARNING - load average: 9.29, 15.10, 10.80 [22:49:03] Load avg. on ortelius is OK: OK - load average: 7.30, 13.54, 10.52 [23:05:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:05:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43686.000000 [23:07:03] aliasd on mayapple is CRITICAL: Connection refused [23:07:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:07:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:08:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 164714 MB (3% inode=67%): [23:08:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:08:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 109899 MB (18% inode=99%): [23:09:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:10:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:18:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16116.000000 [23:21:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58334 MB (5% inode=99%): [23:23:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58840 MB (6% inode=99%): [23:24:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10747 [23:26:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58511 MB (5% inode=99%): [23:28:53] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1941.000000 [23:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:46:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58836 MB (6% inode=99%):