[00:01:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114187.000000 [00:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 344008.000000 [00:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:16:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114194 [00:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78967.000000 [00:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 513140 MB (9% inode=66%): [00:30:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:35:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:42:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88624 MB (14% inode=99%): [00:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:59:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4015.000000 [01:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113925.000000 [01:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 336798.000000 [01:06:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 34651 MB (3% inode=99%): [01:09:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:15:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:16:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113935 [01:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74135.000000 [01:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 512971 MB (9% inode=66%): [01:30:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:34:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:35:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:42:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:51:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86431 MB (14% inode=99%): [01:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [01:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4373.000000 [02:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112522.000000 [02:03:13] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 322423.000000 [02:08:23] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3549.000000 [02:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:16:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111753 [02:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70314.000000 [02:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 511440 MB (9% inode=66%): [02:31:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:34:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:35:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:41:24] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1738.000000 [02:42:32] nacht ts [02:42:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88117 MB (14% inode=99%): [02:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109340.000000 [03:03:13] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 310971.000000 [03:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108589 [03:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67295.000000 [03:26:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493880 MB (9% inode=65%): [03:31:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:35:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:42:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87875 MB (14% inode=99%): [03:51:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 51864 MB (9% inode=99%): [03:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107732.000000 [04:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 310751.000000 [04:09:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:17:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107655 [04:26:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65026.000000 [04:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493154 MB (9% inode=65%): [04:31:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:34:08] Theopolisme * [Toolserver-l] Installing Exiv2: `Make` errors [04:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:35:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:36:55] is theo here? [04:42:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87658 MB (14% inode=99%): [04:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108642.000000 [05:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 305964.000000 [05:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:17:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108837 [05:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63090.000000 [05:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 494009 MB (9% inode=65%): [05:32:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:36:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:42:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87398 MB (14% inode=99%): [05:56:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108263.000000 [06:03:13] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 303241.000000 [06:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107552 [06:21:24] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2080.000000 [06:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61032.000000 [06:26:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493841 MB (9% inode=65%): [06:32:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:35:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:42:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87135 MB (14% inode=99%): [06:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:04] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:58:24] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:58:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:58:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:00:05] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:01:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105407.000000 [07:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 300662.000000 [07:03:24] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:25] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1793.000000 [07:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 105201 [07:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59115.000000 [07:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493620 MB (9% inode=65%): [07:32:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:34:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:34:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86848 MB (14% inode=99%): [07:56:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:15] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104492.000000 [08:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 302346.000000 [08:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:16:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103604 [08:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57482.000000 [08:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493411 MB (9% inode=65%): [08:32:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:34:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86621 MB (14% inode=99%): [08:56:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100747.000000 [09:03:13] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 300225.000000 [09:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 99618 [09:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56328.000000 [09:26:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493229 MB (9% inode=65%): [09:32:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:34:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:34:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85919 MB (14% inode=99%): [09:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:57:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 97932.000000 [10:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 295965.000000 [10:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:11:23] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2004.000000 [10:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 96020 [10:26:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55410.000000 [10:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493013 MB (9% inode=65%): [10:32:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:35:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:44:53] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1905 [10:46:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3604.000000 [10:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85316 MB (14% inode=99%): [10:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:54] MySQL slave on z-dat-s2-b is OK: Uptime: 1251894 Threads: 12 Questions: 1909862235 Slow queries: 17338 Opens: 11331704 Flush tables: 1 Open tables: 256 Queries per second avg: 1525.578 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1709 [11:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93779.000000 [11:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 289988.000000 [11:04:04] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:04:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:15:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92634 [11:26:25] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54013.000000 [11:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 492742 MB (9% inode=65%): [11:32:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:41:54] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1976 [11:42:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1991.000000 [11:46:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3798.000000 [11:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84684 MB (13% inode=99%): [11:54:53] MySQL slave on z-dat-s5-b is OK: Uptime: 1251733 Threads: 4 Questions: 2092465229 Slow queries: 17998 Opens: 1217294 Flush tables: 1 Open tables: 256 Queries per second avg: 1671.654 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1787 [11:55:24] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1743.000000 [11:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89946.000000 [12:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 282288.000000 [12:04:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:12:25] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2074.000000 [12:12:54] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2091 [12:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88755 [12:26:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51931.000000 [12:26:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 511162 MB (9% inode=66%): [12:32:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:46:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5296.000000 [12:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84309 MB (13% inode=99%): [12:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3616.000000 [12:58:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3622 [13:01:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88049.000000 [13:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 280816.000000 [13:04:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:06:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 33633 MB (3% inode=99%): [13:07:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3592.000000 [13:07:53] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3562 [13:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:15:24] wikidata replag on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [13:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86697 [13:26:33] wikidata replag on rosemary is CRITICAL: (Service Check Timed Out) [13:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 508547 MB (9% inode=66%): [13:28:54] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1868 [13:31:53] MySQL slave on z-dat-s2-b is OK: Uptime: 1261134 Threads: 11 Questions: 1931638109 Slow queries: 17449 Opens: 11498389 Flush tables: 1 Open tables: 256 Queries per second avg: 1531.667 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1729 [13:33:45] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:34:54] MySQL slave on z-dat-s5-b is OK: Uptime: 1257733 Threads: 3 Questions: 2094795898 Slow queries: 18070 Opens: 1217988 Flush tables: 1 Open tables: 255 Queries per second avg: 1665.533 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [13:35:24] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1745.000000 [13:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:37:14] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2198.000000 [13:37:24] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2209.000000 [13:37:44] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [13:38:33] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2192.000000 [13:38:34] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2289.000000 [13:45:25] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2030.000000 [13:45:54] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2044 [13:46:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8101.000000 [13:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83953 MB (13% inode=99%): [13:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [14:01:14] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3637.000000 [14:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84436.000000 [14:01:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [14:02:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3633.000000 [14:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 283823.000000 [14:03:24] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [14:03:24] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1779.000000 [14:03:53] MySQL slave on z-dat-s5-b is OK: Uptime: 1259473 Threads: 3 Questions: 2095682380 Slow queries: 18100 Opens: 1218203 Flush tables: 1 Open tables: 256 Queries per second avg: 1663.935 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1768 [14:04:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:15:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84003 [14:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 508335 MB (9% inode=66%): [14:27:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56411.000000 [14:27:54] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1914 [14:28:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1942.000000 [14:31:24] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1751.000000 [14:31:54] MySQL slave on z-dat-s5-b is OK: Uptime: 1261153 Threads: 3 Questions: 2096617205 Slow queries: 18136 Opens: 1218417 Flush tables: 1 Open tables: 255 Queries per second avg: 1662.460 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1738 [14:33:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:34:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:43:08] Daniel Schwen * Re: [Toolserver-l] Installing Exiv2: `Make` errors [14:46:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7019.000000 [14:51:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83644 MB (13% inode=99%): [14:56:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:54] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2095 [14:58:23] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2122.000000 [14:59:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7210.000000 [15:01:14] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7237.000000 [15:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82715.000000 [15:01:43] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7256.000000 [15:02:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7233.000000 [15:03:13] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 282398.000000 [15:04:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:11:53] MySQL slave on z-dat-s5-b is OK: Uptime: 1263553 Threads: 5 Questions: 2097694687 Slow queries: 18180 Opens: 1218565 Flush tables: 1 Open tables: 256 Queries per second avg: 1660.155 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1792 [15:12:24] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [15:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82086 [15:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 507921 MB (9% inode=66%): [15:27:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60011.000000 [15:33:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:46:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7147.000000 [15:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83325 MB (13% inode=99%): [15:51:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 50769 MB (9% inode=99%): [15:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10810.000000 [16:01:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10837.000000 [16:01:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82806.000000 [16:01:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10856.000000 [16:02:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10832.000000 [16:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 283338.000000 [16:04:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:15:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82715 [16:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 507610 MB (9% inode=66%): [16:27:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63611.000000 [16:33:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:36:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:46:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7209.000000 [16:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83053 MB (13% inode=99%): [16:52:48] ptwiki_p logging.log_params seems to have two diferent kinds of outputs: last month logs are like "a:2:{s:12:"4::oldgroups";a:1:{i:0;s:12:"autoreviewer";}s:12:"5::newgroups";a:0:{}}", and older with a diferent formatation wich i dint' understood yet, is this a bug? [16:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14409.000000 [17:01:14] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14437.000000 [17:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81991.000000 [17:01:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14456.000000 [17:02:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14433.000000 [17:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 286938.000000 [17:04:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:09:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:16:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 81657 [17:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 506711 MB (9% inode=66%): [17:27:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67211.000000 [17:34:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:34:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:46:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8021.000000 [17:51:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82778 MB (13% inode=99%): [17:56:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18009.000000 [18:01:14] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18037.000000 [18:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80884.000000 [18:01:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18055.000000 [18:02:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18033.000000 [18:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 290538.000000 [18:04:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:16:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 80165 [18:26:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 506423 MB (9% inode=66%): [18:27:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70812.000000 [18:34:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:34:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:36:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:46:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8584.000000 [18:48:08] Theopolisme * Re: [Toolserver-l] Installing Exiv2: `Make` errors [18:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82500 MB (13% inode=99%): [18:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:57:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21609.000000 [19:01:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21638.000000 [19:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77315.000000 [19:01:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21655.000000 [19:02:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21632.000000 [19:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 294138.000000 [19:04:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77740 [19:22:08] Daniel Schwen * Re: [Toolserver-l] Installing Exiv2: `Make` errors [19:26:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 505962 MB (9% inode=66%): [19:27:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74411.000000 [19:32:08] Theopolisme * Re: [Toolserver-l] Installing Exiv2: `Make` errors [19:34:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:34:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:36:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:45:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:46:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8673.000000 [19:51:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82172 MB (13% inode=99%): [19:53:57] hello [19:54:01] maintenance tonight [19:54:21] todays plan is to renumber the login servers [19:54:39] i think ill start with nightshade [19:54:47] i will go prepare now [19:56:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:57:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25210.000000 [20:01:14] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25238.000000 [20:01:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78478.000000 [20:01:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25255.000000 [20:02:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25233.000000 [20:02:58] going to reboot nightshade [20:03:14] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 297738.000000 [20:04:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:09:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78431 [20:26:03] had no home - no nfs - so ill try if nfs works from the other ha node too now [20:26:09] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:26:26] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:26:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:26:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 505612 MB (9% inode=66%): [20:26:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:27:16] aliasd on yarrow is CRITICAL: Connection refused [20:27:25] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78013.000000 [20:32:19] switching back nfs [20:32:23] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:34:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:35:18] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:46:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:46:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7653.000000 [20:52:17] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81920 MB (13% inode=99%): [20:52:18] nightshade ask for password, it didn't restart correctly ? [20:53:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:57:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:57:49] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:46] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:59:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:59:46] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28813.000000 [21:01:16] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28841.000000 [21:01:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78223.000000 [21:01:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28859.000000 [21:01:46] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:02:37] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28841.000000 [21:03:37] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 301355.000000 [21:09:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:10:07] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:10:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:18:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78074 [21:27:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 505205 MB (9% inode=66%): [21:28:07] aliasd on yarrow is CRITICAL: Connection refused [21:28:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81665.000000 [21:28:15] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2220.000000 [21:29:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:35:47] phe: still having issues accessing ldap [21:36:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:36:07] phe: so login into nightshade is broken [21:37:54] nosy, sge can submit job to nightshade ? [21:38:26] oh..better not without sge working...ok ill disable nightshade in sge... [21:39:06] / on hemlock is WARNING: DISK WARNING - free space: / 2867 MB (14% inode=87%): [21:39:25] /tmp on hemlock is WARNING: DISK WARNING - free space: / 2804 MB (14% inode=87%): [21:40:56] phe: sge was already disabled for nightshade [21:42:25] /tmp on hemlock is CRITICAL: DISK CRITICAL - free space: / 2145 MB (10% inode=87%): [21:43:06] / on hemlock is CRITICAL: DISK CRITICAL - free space: / 1970 MB (9% inode=87%): [21:47:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11097.000000 [21:48:16] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: / 825 MB (4% inode=87%): [21:52:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3660.000000 [21:52:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81642 MB (13% inode=99%): [21:54:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:58:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:58:44] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32458.000000 [22:02:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32491.000000 [22:02:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78550.000000 [22:02:25] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32504.000000 [22:03:15] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32480.000000 [22:04:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 304989.000000 [22:10:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:10:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:12:22] Hello all [22:16:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:18:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78073 [22:27:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 504931 MB (9% inode=66%): [22:28:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85264.000000 [22:29:06] aliasd on yarrow is CRITICAL: Connection refused [22:30:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:36:16] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:42:25] /tmp on hemlock is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=87%): [22:44:06] / on hemlock is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=87%): [22:47:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14698.000000 [22:48:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=87%): [22:52:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81377 MB (13% inode=99%): [22:53:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7320.000000 [22:54:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:57:15] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:35] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:35] NTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:35] /tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:44] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:44] Environment IPMI on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:54] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:35] Sensors on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:35] / on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:35] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:58:44] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36058.000000 [23:02:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36091.000000 [23:02:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77307.000000 [23:02:28] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36104.000000 [23:03:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36081.000000 [23:04:44] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 308622.000000 [23:10:06] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [23:10:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:10:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:16:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:18:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77046 [23:20:21] DaBPunkt, is SGE enable on nightshade now ? [23:23:54] MySQL slave on z-dat-s2-b is OK: Uptime: 44 Threads: 6 Questions: 2608 Slow queries: 1 Opens: 4058 Flush tables: 1 Open tables: 183 Queries per second avg: 59.272 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1303 [23:27:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 503452 MB (9% inode=66%): [23:28:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88865.000000 [23:29:07] aliasd on yarrow is CRITICAL: Connection refused [23:30:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:30:46] it's disable atm [23:37:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:37:36] / on damiana is WARNING: DISK WARNING - free space: / 13988 MB (19% inode=95%): [23:38:53] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1991 [23:42:12] phe: I'm i the middle of the database-maintenance. I have no idea what nosy did today. [23:42:24] /tmp on hemlock is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=87%): [23:44:06] / on hemlock is CRITICAL: DISK CRITICAL - free space: / 64 MB (0% inode=87%): [23:47:06] SMTP on nightshade is OK: SMTP OK - 0.046 sec. response time [23:47:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18294.000000 [23:47:15] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:47:15] Environment IPMI on nightshade is OK: ok: temperature ok fan ok voltage ok chassis ok [23:47:24] Load avg. on nightshade is OK: OK - load average: 0.30, 0.10, 0.03 [23:47:24] NTP on nightshade is OK: NTP OK: Offset -0.004366 secs [23:47:54] MySQL slave on z-dat-s2-b is OK: Uptime: 1485 Threads: 6 Questions: 711133 Slow queries: 42 Opens: 22034 Flush tables: 1 Open tables: 256 Queries per second avg: 478.877 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1777 [23:48:08] Sensors on nightshade is OK: sensor ok [23:48:08] / on nightshade is OK: DISK OK - free space: / 1593 MB (89% inode=94%): [23:48:08] aliasd on nightshade is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [23:48:08] /tmp on nightshade is OK: DISK OK - free space: /tmp 4317 MB (96% inode=99%): [23:48:08] / on hemlock is OK: DISK OK - free space: / 4757 MB (23% inode=87%): [23:48:15] /aux0 on hemlock is OK: DISK OK - free space: / 4757 MB (23% inode=87%): [23:48:24] /tmp on hemlock is OK: DISK OK - free space: / 4757 MB (23% inode=87%): [23:52:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81063 MB (13% inode=99%): [23:53:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10920.000000 [23:54:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:58:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:58:44] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:07] Sun Grid Engine execd on nightshade is CRITICAL: CRITICAL: execd not communicating