[00:01:15] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:11:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98629.000000 [00:14:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6829.000000 [00:15:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5342.000000 [00:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:16:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:17:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98761 [00:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69069.000000 [00:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 482558 MB (9% inode=65%): [00:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:35:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3872.000000 [00:43:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:46:04] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3598.000000 [00:50:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48626.000000 [00:54:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69215 MB (11% inode=99%): [01:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:04] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31345 MB (3% inode=99%): [01:11:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99667.000000 [01:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5215.000000 [01:15:04] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3597.000000 [01:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:17:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:20:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 100144 [01:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69426.000000 [01:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 482552 MB (9% inode=65%): [01:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:46:04] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1807.000000 [01:47:04] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1778.000000 [01:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49014.000000 [01:53:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67058 MB (10% inode=99%): [02:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99815.000000 [02:14:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3924.000000 [02:15:03] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1833.000000 [02:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:17:03] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1772.000000 [02:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:20:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 100130 [02:24:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69769 MB (11% inode=99%): [02:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68764.000000 [02:32:44] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3582.000000 [02:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 480630 MB (9% inode=65%): [02:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48104.000000 [03:01:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:03:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99234.000000 [03:15:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 99191 [03:24:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68450 MB (11% inode=99%): [03:31:43] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68268.000000 [03:32:44] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2411.000000 [03:33:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 481703 MB (9% inode=65%): [03:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:41:24] I got a mail saying "Your job 1690381 is was running on host yarrow.toolserver.org. Pushed manually/automatic rescheduling for this job.". what does this mean? [03:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48636.000000 [03:56:37] liangent: Was that at the time of the reboots? [03:59:05] scfc_de: dunno. I know nothing about any reboot [04:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:01:31] Happened about 20:32Z. I don't know what the contents of "qacct -j 1690381" mean, but it should have been a singular event in any case. [04:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100716.000000 [04:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:17:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 101106 [04:24:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68271 MB (11% inode=99%): [04:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69002.000000 [04:32:43] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3454.000000 [04:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:34:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 481294 MB (9% inode=65%): [04:37:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3611.000000 [04:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:44:43] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3577.000000 [04:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48288.000000 [05:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:09:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3611.000000 [05:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103023.000000 [05:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:20:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103137 [05:24:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68088 MB (11% inode=99%): [05:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71120.000000 [05:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:34:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 480517 MB (9% inode=65%): [05:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49761.000000 [06:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:03:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:09:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4323.000000 [06:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103624.000000 [06:15:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103629 [06:24:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67880 MB (11% inode=99%): [06:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73288.000000 [06:33:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:35:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 481393 MB (9% inode=65%): [06:43:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50470.000000 [07:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3669.000000 [07:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104603.000000 [07:11:44] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3591.000000 [07:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 104706 [07:24:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67621 MB (11% inode=99%): [07:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75564.000000 [07:33:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:35:05] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 481472 MB (9% inode=65%): [07:43:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:50:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51313.000000 [08:00:43] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1785.000000 [08:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104796.000000 [08:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:17:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 104898 [08:24:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67381 MB (11% inode=99%): [08:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78550.000000 [08:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:35:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 481276 MB (9% inode=65%): [08:43:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51746.000000 [09:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:24] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.591 second response time [09:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:14] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.622 second response time [09:10:23] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 9.053 second response time [09:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105486.000000 [09:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:20:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 105753 [09:23:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67092 MB (10% inode=99%): [09:27:13] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.337 second response time [09:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81448.000000 [09:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:35:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 481130 MB (9% inode=65%): [09:44:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:45:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 46306 MB (8% inode=99%): [09:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51767.000000 [09:55:14] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2186.000000 [10:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:03:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:03:24] Load avg. on nightshade is WARNING: WARNING - load average: 14.76, 16.11, 13.41 [10:05:24] Load avg. on nightshade is OK: OK - load average: 11.10, 14.24, 13.04 [10:11:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106446.000000 [10:15:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:20:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106949 [10:23:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66757 MB (10% inode=99%): [10:26:36] [[Special:Log/newusers]] create 10 * Fcarbonara * (New user account) [10:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84519.000000 [10:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:35:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 480641 MB (9% inode=65%): [10:44:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:45:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3640.000000 [10:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52488.000000 [11:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106747.000000 [11:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106232 [11:23:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66440 MB (10% inode=99%): [11:26:30] @replag [11:29:44] DarkoNeko: s1-rr-a-wd: 15h 5m 25s [+0.28 s/s]; s1-user: 1d 5h 25m 32s [+0.10 s/s]; s1-user-c: 1h 43m 29s [+0.01 s/s]; s1-user-wd: 1d 25m 52s [+0.24 s/s]; s2-rr: 20m 22s [+0.00 s/s]; s2-user: 20m 22s [+0.00 s/s]; s2-user-c: error; s2-user-wd: 15h 5m 26s [+0.28 s/s] [11:29:45] DarkoNeko: s3-user: 14s [-0.00 s/s]; s3-user-wd: 2h 4m 23s [-0.01 s/s]; s5-user: 2m 9s [+0.00 s/s] [11:31:30] uwa [11:31:43] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88094.000000 [11:33:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:35:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 480473 MB (9% inode=65%): [11:44:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:46:14] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7097.000000 [11:50:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55181.000000 [12:00:26] @replag [12:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:42] liangent: s1-rr-a-wd: 15h 29m 2s [+0.70 s/s]; s1-user: 1d 5h 34m 40s [+0.27 s/s]; s1-user-c: 2h 11m 45s [+0.83 s/s]; s1-user-wd: 1d 58m 52s [+0.97 s/s]; s2-rr: 19m 27s [-0.03 s/s]; s2-user: 19m 27s [-0.03 s/s]; s2-user-c: error; s2-user-wd: 15h 29m 6s [+0.70 s/s] [12:03:43] liangent: s3-user: 34s [+0.01 s/s]; s3-user-wd: 1h 43m 57s [-0.60 s/s]; s4-user-wd: 20s [-0.03 s/s]; s5-user: 1m 34s [-0.02 s/s]; s5-user-c: 13s [-0.44 s/s] [12:12:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106863.000000 [12:15:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:17:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107292 [12:23:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66142 MB (10% inode=99%): [12:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91614.000000 [12:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:35:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 480184 MB (9% inode=65%): [12:44:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:46:14] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10280.000000 [12:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57512.000000 [13:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:36] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:03:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:04] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 28784 MB (2% inode=99%): [13:12:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108753.000000 [13:16:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:20:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108545 [13:23:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65707 MB (10% inode=99%): [13:31:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95201.000000 [13:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:35:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 479414 MB (9% inode=64%): [13:37:26] [[Operators]] 10https://wiki.toolserver.org/w/index.php?diff=7892&oldid=7869&rcid=21727 * Dab * (+148) (+List of Operators) [13:37:59] [[Special:Log/rights]] rights 10 * Dab * (changed group membership for User:Apmon from confluence-users, jira-users to confluence-users, jira-users, mw-bureaucrats: Opeartor) [13:38:46] Hello all [13:41:14] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1633 MB (12% inode=99%): [13:44:14] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1157 MB (9% inode=99%): [13:44:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:46:24] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13814.000000 [13:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59755.000000 [13:50:14] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1477 MB (11% inode=99%): [14:01:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:03:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:14] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1033 MB (8% inode=99%): [14:13:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107922.000000 [14:13:08] DaB. * [Toolserver-announce] Adding of 2 operators [14:16:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:17:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:20:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108077 [14:23:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65370 MB (10% inode=99%): [14:31:43] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98592.000000 [14:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:35:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 478703 MB (8% inode=64%): [14:38:22] @replag [14:41:12] we seem to be suffering from a replag-lag :-( [14:41:34] russblau: s1-rr-a-wd: 16h 51m 54s [+0.52 s/s]; s1-user: 1d 6h 7m 48s [+0.21 s/s]; s1-user-c: 4h 27m 31s [+0.86 s/s]; s1-user-wd: 1d 3h 31m 52s [+0.97 s/s]; s2-rr: 1m 41s [-0.11 s/s]; s2-user: 1m 41s [-0.11 s/s]; s2-user-c: error; s2-user-wd: 16h 51m 54s [+0.52 s/s] [14:41:35] russblau: s3-user-wd: 1h 28m 5s [-0.10 s/s] [14:42:19] * closedmouth punches the toolserver [14:43:02] :( [14:43:12] even though we now have two s1 servers again, the situation just gets worse and worse [14:44:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:46:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:47:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16262.000000 [14:50:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61018.000000 [14:56:38] russblau: ithe replag-lag will properly vanish when we find the ipv6-problem with z-dat-s1-b [14:58:32] DaBPunkt: thanks for the info. I'm sure this is even more frustrating for you than it is for us users [15:00:56] russblau: yes and no [15:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:02:32] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:02:56] it sucks to have no spare-server somewhere for testing or to replace a broken server – or having not enough server in general. But I have not much tools of my own so it is not so frustrating for me as for the tool-authors [15:03:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:22] /tmp on ortelius is OK: DISK OK - free space: /tmp 8286 MB (68% inode=99%): [15:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109554.000000 [15:34:13] [15:37:13] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2133.000000 [16:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 478317 MB (8% inode=64%): [16:01:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:01:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19927.000000 [16:01:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62337.000000 [16:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103776.000000 [16:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64946 MB (10% inode=99%): [16:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110657 [16:01:34] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:02:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:03:34] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [16:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 111088.000000 [16:20:10] DaBPunkt: Hi [17:00:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:00:52] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 477892 MB (8% inode=64%): [17:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:01:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23238.000000 [17:01:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62512.000000 [17:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107329.000000 [17:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113112 [17:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64748 MB (10% inode=99%): [17:01:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:01:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:02:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:04:34] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6086.000000 [17:13:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113616.000000 [18:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 477618 MB (8% inode=64%): [18:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:01:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63924.000000 [18:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110549.000000 [18:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64545 MB (10% inode=99%): [18:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 115417 [18:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:02:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24521.000000 [18:02:32] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:02:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8865.000000 [18:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115958.000000 [18:31:53] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 477300 MB (8% inode=64%): [19:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63753.000000 [19:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 117502 [19:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64172 MB (10% inode=99%): [19:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113842.000000 [19:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:01:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:02:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26925.000000 [19:02:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:03:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11121.000000 [19:13:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117564.000000 [19:18:47] apmon: pong [19:26:00] what do you need information wise? [19:28:07] apmon: your IRC-nick seems not be registrated, so I can not give you OP-rights; but that's only a minor problem. What I need is a htpass-file so I can give our access to pupet's SVN [19:28:32] give our → give you [19:29:00] Oh, I think it is registered on oftc, but not freenode. I can do that on freenode as well [19:31:36] perhaps a silly question, but how do I log in as operator? [19:32:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:32:33] apmon: change the login-name to oapmon [19:33:03] does it use the same password? [19:33:13] the same key [19:33:30] whcih host do you try? [19:34:43] willow [19:34:57] apmon: it works only on linux-hosts [19:36:10] ah, indeed, on nightshade it works [19:36:57] creating the htpass file now [19:44:14] I have created a myhtpasswordfile in my homedir [19:45:31] next time make it not world-readable ;) [19:45:59] apmon: it is not salted [19:46:10] which command did you used? [19:46:58] htpasswd -c myhtpasswordfile oapmon on nightshade [19:48:22] I assumed that would handle things correctly, but was wondering why the hash looked so short [19:50:45] mm, strange [19:50:53] Let me look [19:55:20] apmon: it is salted, but the salt is quite short afais. Please re-try with the "-m" option to get a longer salt [19:59:48] Ok, I created myhtpasswordfile3 [20:00:32] looks good [20:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 476947 MB (8% inode=64%): [20:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66173.000000 [20:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118727 [20:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64000 MB (10% inode=99%): [20:01:33] Is there a procedure / documentation of how to get packages into the toolserver repository? [20:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117078.000000 [20:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:01:39] apmon: https://svn.toolserver.org/svnroot/system/puppet/ is the path of the svn-repository [20:01:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:02:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28977.000000 [20:02:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:03:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14064.000000 [20:06:44] I had checked out the puppet repository (anonymously) and starting having a look over it yesterday [20:13:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118733.000000 [20:15:18] I'll try and set up a local debian stable VM and try and create the necessary debian packages for osm2pgsql, mod_tile and tirex [20:15:37] or at least figure out which packages need back-porting [20:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:35:59] DaBPunkt: I have a problem with a tool of erwin, but I am wondering if it might be indicating database problems, have you seen any indication? [20:36:20] I use the tool on nl-wikipedia, which is S2 I believe [20:38:24] s2 looks good for m4e [20:39:32] DaBPunkt: hmm, ok [20:39:49] I will ask Erwin then [20:44:27] DaBPunkt: I sent the server proposal to Silke [20:44:30] apmon: if you can create a debian package-list (it is a bit tricky) it should be possible to coy everthing in a subfolder of your operator-account and include it in the source-file of the osm-server [21:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 476577 MB (8% inode=64%): [21:01:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:01:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66942.000000 [21:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119657 [21:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63849 MB (10% inode=99%): [21:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120345.000000 [21:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:02:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31166.000000 [21:02:32] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:03:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15487.000000 [21:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119879.000000 [21:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:46:33] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 45440 MB (8% inode=99%): [21:48:12] Which command opens a new screen sesion? I don't remember [22:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 476257 MB (8% inode=64%): [22:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66647.000000 [22:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120587 [22:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63653 MB (10% inode=99%): [22:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123058.000000 [22:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:01:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:02:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30990.000000 [22:02:32] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:06:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13611.000000 [22:13:21] hi [22:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121120.000000 [22:13:23] maybe it is wrong place but I so not know where is correct [22:13:34] I cannot access my JIRA account [22:14:38] I create it two weeks ago and put issue for ts account request [22:14:57] today I decide to check it [22:15:10] I put my username and password, but got message 'Sorry, your username and password are incorrect - please try again.' [22:15:11] When I try restore password by username or email I got message 'No user with that username exists.' [22:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:42:52] Ahonc: are you sure you use the correct name? [22:44:17] yes. I copypaste It from e-mail [22:44:50] whichh name? [22:44:52] or tried use this URL: https://jira.toolserver.org/secure/ForgotLoginDetails.jspa?username=ahonc [22:44:53] Load avg. on ortelius is WARNING: WARNING - load average: 17.19, 15.88, 12.09 [22:45:05] got error too [22:45:16] username is ahonc [22:45:45] I hate that part of jira… [22:47:53] Load avg. on ortelius is OK: OK - load average: 13.73, 14.77, 12.33 [22:50:25] Ahonc: try to register again with the same name [22:57:00] I'd reregister [22:57:07] it works now [22:59:32] good [23:00:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 476019 MB (8% inode=64%): [23:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:01:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66930.000000 [23:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123076 [23:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63413 MB (10% inode=99%): [23:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123929.000000 [23:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:02:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28919.000000 [23:02:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:06:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11187.000000 [23:12:06] nacht ts [23:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122663.000000 [23:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output