[00:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8219 [00:27:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11048.000000 [00:28:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8158.000000 [00:31:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:31:51] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 185401.000000 [00:33:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:33:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 413394.000000 [00:33:51] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:34:21] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 712795.000000 [00:34:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1182029.000000 [00:35:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:35:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1171214.000000 [00:35:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:35:30] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:35:30] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [00:35:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:35:41] aliasd on yarrow is CRITICAL: Connection refused [00:35:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:36:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41914 MB (6% inode=99%): [00:36:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:36:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:36:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:36:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:37:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:37:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:37:30] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:37:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:37:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:37:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:37:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:37:53] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:37:53] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:38:01] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:38:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:38:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:38:22] aliasd on mayapple is CRITICAL: No data received from host [00:38:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:39:11] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:39:11] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:39:11] / on thyme is UNKNOWN: NRPE: Unable to read output [00:39:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:39:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:39:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:39:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:39:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:39:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:40:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359513 MB (6% inode=62%): [00:40:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:40:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:24] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:51] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:11:40] @replag [01:11:43] Betacommand: s1-rr-a: 2d 4h 8m 39s [+1.00 s/s]; s1-rr-a-wd: 6d 9h 21m 37s [+0.71 s/s]; s1-user: 2h 5m 11s [+0.07 s/s]; s1-user-c: 2h 22m 43s [+0.08 s/s]; s1-user-wd: 1w 6d 13h 33m 49s [+0.43 s/s]; s2-rr: 4m 43s [-2.04 s/s]; s2-user: 4m 43s [-2.04 s/s]; s2-user-c: error [01:11:44] Betacommand: s2-user-wd: 1w 1d 6h 11m 22s [+0.33 s/s]; s3-user: 12s [-0.00 s/s]; s3-user-wd: 6d 19h 46m 26s [-0.11 s/s]; s4-user-wd: 1w 6d 16h 55m 17s [+0.94 s/s]; s5-rr-a-wd: 4d 18h 51m 25s [+0.29 s/s]; s5-user: 2d 4h 8m 52s [+1.00 s/s]; s6-user-wd: 1w 19h 56m 56s [-0.08 s/s]; s7-user-wd: 6d 11h 34m 25s [+0.26 s/s] [01:16:04] John * [Toolserver-l] Replag [01:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7315 [01:27:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8220.000000 [01:28:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7194.000000 [01:31:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:31:52] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 413863.000000 [01:33:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:34:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 189061.000000 [01:34:01] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:01] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:34:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 714133.000000 [01:34:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1185447.000000 [01:35:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:35:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1172911.000000 [01:35:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:35:30] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:35:30] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [01:35:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:35:42] aliasd on yarrow is CRITICAL: Connection refused [01:35:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:36:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41735 MB (6% inode=99%): [01:36:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:36:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:36:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:36:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:37:23] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:37:30] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:37:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:37:56] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:37:56] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:37:56] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:37:56] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:37:56] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:38:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:38:22] aliasd on mayapple is CRITICAL: No data received from host [01:38:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:39:01] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:39:01] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:39:11] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:39:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:39:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:39:12] / on thyme is UNKNOWN: NRPE: Unable to read output [01:39:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:39:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:39:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:40:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 355006 MB (6% inode=62%): [01:40:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:40:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:24] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:24] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:14:02] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3392.000000 [02:24:01] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1602.000000 [02:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6397 [02:28:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6231.000000 [02:31:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:32:52] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 415268.000000 [02:33:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:34:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 192661.000000 [02:34:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:34:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:34:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 714754.000000 [02:35:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:35:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1174475.000000 [02:35:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1188959.000000 [02:35:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:36:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:36:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42163 MB (6% inode=99%): [02:36:30] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [02:36:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [02:36:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:36:42] aliasd on yarrow is CRITICAL: Connection refused [02:36:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:37:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:37:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:37:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:30] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:37:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:37:30] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:37:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:37:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:37:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:37:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:37:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:37:53] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:38:23] aliasd on mayapple is CRITICAL: No data received from host [02:38:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:39:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:39:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:39:11] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:39:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:39:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:39:11] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:39:11] / on thyme is UNKNOWN: NRPE: Unable to read output [02:39:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:39:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:39:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:11] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:40:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 354781 MB (6% inode=62%): [02:40:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:40:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:24] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:51] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:49] [[Special:Log/newusers]] create 10 * Nongqua2 * (New user account) [02:49:06] [[User talk:Nongqua2]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7936&rcid=21802 * Nongqua2 * (+538) (Created page with "With broad comprehension and our skills on the franchise market place of Australia, we are very proud of belonging to the best franchise consulting firms in the Oceania zone. As ...") [03:04:07] Susan: ^ [03:04:41] [[Special:Log/delete]] delete 10 * Legoktm * (deleted "[[02User talk:Nongqua210]]": Spam) [03:04:47] Hi closedmouth. [03:04:48] I'm here. [03:05:13] [[Special:Log/block]] block 10 * Legoktm * (blocked [[02User:Nongqua210]] with an expiry time of infinite (account creation disabled, e-mail blocked): Spam) [03:09:22] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3588 [03:09:52] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3551.000000 [03:25:25] oh hi legoktm [03:25:35] o/ [03:29:22] MySQL slave on rosemary is OK: Uptime: 549525 Threads: 17 Questions: 379801251 Slow queries: 204155 Opens: 33332 Flush tables: 1 Open tables: 3634 Queries per second avg: 691.144 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [03:29:52] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1739.000000 [03:31:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:32:52] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 417529.000000 [03:33:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:34:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 196261.000000 [03:34:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:34:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:34:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 716613.000000 [03:35:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:35:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1176629.000000 [03:35:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1192433.000000 [03:35:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:36:01] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:36:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41568 MB (6% inode=99%): [03:36:31] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [03:36:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [03:36:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:36:42] aliasd on yarrow is CRITICAL: Connection refused [03:36:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:37:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:37:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:37:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:35] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:37:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:37:35] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:37:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:37:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:37:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:37:53] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:37:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:37:53] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:38:23] aliasd on mayapple is CRITICAL: No data received from host [03:38:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:39:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:39:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:39:11] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:39:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:39:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:39:11] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:39:12] / on thyme is UNKNOWN: NRPE: Unable to read output [03:39:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:39:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:39:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:40:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:40:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 354584 MB (6% inode=62%): [03:40:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:41:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:41:21] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:41:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:02] Free Memory on turnera is WARNING: WARNING - 6.3% (531732 kB) free! [03:59:02] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (382912 kB) free! [04:00:02] Free Memory on turnera is WARNING: WARNING - 6.3% (524156 kB) free! [04:05:01] Free Memory on turnera is OK: OK - 7.4% (616156 kB) free. [04:31:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:32:51] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:34:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 199861.000000 [04:34:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:34:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 420238.000000 [04:34:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 718982.000000 [04:35:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:35:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1179333.000000 [04:35:52] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:35:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1195985.000000 [04:35:52] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2198.000000 [04:36:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:36:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41499 MB (6% inode=99%): [04:36:30] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [04:36:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [04:36:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:36:42] aliasd on yarrow is CRITICAL: Connection refused [04:36:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:37:23] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:37:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:37:52] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:37:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:37:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:37:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:38:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:38:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:38:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:38:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:38:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:38:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:39:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:39:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:39:11] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:39:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:39:12] / on thyme is UNKNOWN: NRPE: Unable to read output [04:39:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:39:22] aliasd on mayapple is CRITICAL: No data received from host [04:39:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:39:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:39:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:39:53] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:39:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:40:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:40:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:40:23] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 354458 MB (6% inode=62%): [04:41:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:41:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:41:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:57:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2173.000000 [04:59:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [05:21:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [05:31:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:33:52] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:33:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:34:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 423837.000000 [05:34:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 722582.000000 [05:35:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 203521.000000 [05:35:01] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:35:01] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:35:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:35:52] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:35:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1199585.000000 [05:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:36:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1182994.000000 [05:36:30] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:36:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [05:36:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:36:46] aliasd on yarrow is CRITICAL: Connection refused [05:37:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:37:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41404 MB (6% inode=99%): [05:37:51] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:37:51] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:38:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:38:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:38:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:38:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:38:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:38:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:38:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:38:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:38:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:38:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:39:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:39:12] / on thyme is UNKNOWN: NRPE: Unable to read output [05:39:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:39:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:39:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:39:30] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:39:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:39:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:39:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:01] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:22] aliasd on mayapple is CRITICAL: No data received from host [05:40:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:40:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:41:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:41:21] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 354214 MB (6% inode=62%): [05:42:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:42:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:42:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:42:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:42:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:42:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:42:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:42:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:42:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:50:01] [[Special:Log/newusers]] create 10 * Germinal * (New user account) [05:59:51] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7238.000000 [06:22:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7272.000000 [06:25:01] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (368900 kB) free! [06:28:01] Free Memory on turnera is WARNING: WARNING - 5.9% (494644 kB) free! [06:31:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:32:22] SSH on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:02] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:02] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:34:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 726182.000000 [06:35:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207121.000000 [06:35:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:35:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:35:02] Free Memory on turnera is OK: OK - 7.5% (631028 kB) free. [06:35:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:35:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 427498.000000 [06:35:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1203186.000000 [06:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:36:02] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:36:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1186594.000000 [06:36:30] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:36:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:36:53] aliasd on yarrow is CRITICAL: Connection refused [06:37:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:37:30] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:38:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:38:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41229 MB (6% inode=99%): [06:38:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:38:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:38:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:38:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:39:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:39:02] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:39:02] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:39:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:39:02] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:39:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:39:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:39:30] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:39:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:40:01] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:02] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:02] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:40:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:40:12] / on thyme is UNKNOWN: NRPE: Unable to read output [06:41:22] aliasd on mayapple is CRITICAL: No data received from host [06:41:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:41:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353981 MB (6% inode=62%): [06:41:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:41:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:42:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:42:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:42:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:42:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:42:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:42:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:42:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:42:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:43:01] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:41] aliasd on nightshade is CRITICAL: Connection refused [06:52:22] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:52:41] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:52:51] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:56:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:56:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:58:01] Free Memory on turnera is WARNING: WARNING - 6.7% (559228 kB) free! [06:58:41] aliasd on nightshade is OK: TCP OK - 0.005 second response time on port 984 [500 Not found.] [06:59:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7957.000000 [07:00:01] Free Memory on turnera is OK: OK - 7.5% (632628 kB) free. [07:02:02] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 104764 MB (10% inode=99%): [07:05:41] aliasd on nightshade is CRITICAL: Connection refused [07:15:02] Free Memory on turnera is WARNING: WARNING - 6.7% (562616 kB) free! [07:18:23] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:41] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:19:51] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:20:27] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:20:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:21:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:22:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3570.000000 [07:31:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:32:22] SSH on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:33:52] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1757.000000 [07:34:02] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:02] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:34:31] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 727701.000000 [07:35:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208591.000000 [07:35:01] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:01] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:35:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1206525.000000 [07:36:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:36:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 428904.000000 [07:36:30] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 208487 [07:36:41] aliasd on nightshade is CRITICAL: Connection refused [07:37:02] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:37:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1187878.000000 [07:37:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:37:31] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 203720 [07:37:52] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:37:52] aliasd on yarrow is CRITICAL: Connection refused [07:38:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41058 MB (6% inode=99%): [07:38:30] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:38:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:39:01] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:39:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:39:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:39:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:39:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:39:30] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:39:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:39:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:40:01] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:40:01] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:01] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:02] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:02] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:03] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:03] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:11] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:11] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:41:11] / on thyme is UNKNOWN: NRPE: Unable to read output [07:41:22] aliasd on mayapple is CRITICAL: No data received from host [07:41:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353802 MB (6% inode=62%): [07:41:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:41:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:41:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:42:21] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:42:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:42:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:42:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:42:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:42:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:42:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:42:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:43:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:00:51] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5815.000000 [08:15:22] SSH on sage is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [08:21:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:21:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:27:04] Calak * [Toolserver-l] Account [08:27:04] Calak * [Toolserver-l] Account [08:31:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:34:01] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:02] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:34:31] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 727192.000000 [08:35:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 204473.000000 [08:35:01] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:01] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:36:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:36:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 429268.000000 [08:36:31] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 204478 [08:36:41] aliasd on nightshade is CRITICAL: Connection refused [08:36:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1209896.000000 [08:37:01] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:37:02] Free Memory on turnera is WARNING: WARNING - 5.2% (436524 kB) free! [08:37:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:37:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1187873.000000 [08:37:03] K. Peachey * Re: [Toolserver-l] Account [08:37:30] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 195868 [08:37:52] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:37:52] aliasd on yarrow is CRITICAL: Connection refused [08:38:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40848 MB (6% inode=99%): [08:38:30] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:38:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:39:01] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:39:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:39:21] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:39:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:39:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:39:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:39:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:40:01] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:40:01] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:01] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:11] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:40:11] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:40:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:40:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:41:02] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:02] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:02] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:02] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:41:12] / on thyme is UNKNOWN: NRPE: Unable to read output [08:41:22] aliasd on mayapple is CRITICAL: No data received from host [08:41:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353397 MB (6% inode=62%): [08:41:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:41:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:31] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:42:01] Free Memory on turnera is OK: OK - 7.2% (606028 kB) free. [08:42:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:42:31] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:43:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:43:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:43:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:43:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:43:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:43:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:50:02] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422772 kB) free! [08:55:41] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:56:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:00:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4677.000000 [09:19:53] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3578.000000 [09:21:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:23:53] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1978.000000 [09:24:52] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1937 [09:27:52] MySQL slave on rosemary is OK: Uptime: 571033 Threads: 11 Questions: 395628629 Slow queries: 209795 Opens: 34158 Flush tables: 1 Open tables: 3640 Queries per second avg: 692.829 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1733 [09:27:52] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1724.000000 [09:30:53] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2197.000000 [09:31:02] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2207 [09:34:02] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:02] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:34:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 729162.000000 [09:35:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 194427.000000 [09:35:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:36:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:36:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 429058.000000 [09:36:52] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 194022 [09:36:52] aliasd on nightshade is CRITICAL: Connection refused [09:36:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1213198.000000 [09:37:02] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:37:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:37:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1187089.000000 [09:37:53] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 184614 [09:37:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:37:53] aliasd on yarrow is CRITICAL: Connection refused [09:38:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40618 MB (6% inode=99%): [09:38:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:39:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:39:21] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:52] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:39:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:40:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:40:02] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:02] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:40:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:40:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:40:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:40:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:40:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:41:01] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:02] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:02] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:03] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:03] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:41:12] / on thyme is UNKNOWN: NRPE: Unable to read output [09:41:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:41:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:52] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353055 MB (6% inode=62%): [09:41:52] aliasd on mayapple is CRITICAL: No data received from host [09:42:02] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:42:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:02] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:52] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:49:00] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1784.000000 [09:56:00] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:56:51] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [09:57:20] MySQL slave on daphne is OK: Uptime: 569440 Threads: 60 Questions: 77964309 Slow queries: 118861 Opens: 24970 Flush tables: 1 Open tables: 805 Queries per second avg: 136.914 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [10:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:02:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 588663.000000 [10:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:29:50] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 16623 MB (1% inode=99%): [10:34:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 187142.000000 [10:35:48] IRC Bot [10:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731473.000000 [10:36:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:36:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 429330.000000 [10:36:31] @replag [10:36:32] Burnley: s1-rr-a: 2d 34m 5s [-]; s1-rr-a-wd: 6d 16h 30m 45s [-]; s1-user-wd: 1w 6d 17h 29m 16s [-]; s2-user-c: error; s2-user-wd: 1w 1d 11h 11m 12s [-]; s3-user: 43s [-]; s3-user-wd: 6d 21h 56m 24s [-]; s4-user-wd: 2w 1h 56m 5s [-] [10:36:33] Burnley: s5-rr-a-wd: 4d 23h 15m 40s [-]; s5-user: 2d 3h 58m 53s [-]; s5-user-c: 22m 28s [-]; s6-user-wd: 1w 21h 28m 49s [-]; s7-user-wd: 6d 19h 58m 57s [-] [10:37:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1186142.000000 [10:37:51] aliasd on nightshade is CRITICAL: Connection refused [10:37:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 187107 [10:37:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1216635.000000 [10:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40375 MB (6% inode=99%): [10:38:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:38:50] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 174423 [10:38:50] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:38:50] aliasd on yarrow is CRITICAL: Connection refused [10:39:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:39:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:40:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:10] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:40:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:41:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:10] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:11] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:41:21] / on thyme is UNKNOWN: NRPE: Unable to read output [10:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:41:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:42:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352703 MB (6% inode=62%): [10:42:51] aliasd on mayapple is CRITICAL: No data received from host [10:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:44:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:02:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 591679.000000 [11:15:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 6290 MB (20% inode=93%): [11:15:50] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6289 MB (20% inode=93%): [11:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:28:00] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1878.000000 [11:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:11] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177148.000000 [11:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 733539.000000 [11:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:36:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 431153.000000 [11:37:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1187583.000000 [11:37:50] aliasd on nightshade is CRITICAL: Connection refused [11:37:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 176776 [11:37:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1220126.000000 [11:38:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40118 MB (6% inode=99%): [11:38:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:38:51] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 163735 [11:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:39:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:50] aliasd on yarrow is CRITICAL: Connection refused [11:39:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:40:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:11] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:40:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:41:10] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:11] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:41:21] / on thyme is UNKNOWN: NRPE: Unable to read output [11:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:41:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:42:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352365 MB (6% inode=62%): [11:42:50] aliasd on mayapple is CRITICAL: No data received from host [11:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:11] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:29] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:29] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:56:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:01:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:02:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 594955.000000 [12:15:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 6008 MB (20% inode=93%): [12:15:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6004 MB (20% inode=93%): [12:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:23:59] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [12:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:35:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 168183.000000 [12:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 736339.000000 [12:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:37:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1190306.000000 [12:37:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 434017.000000 [12:37:50] aliasd on nightshade is CRITICAL: Connection refused [12:37:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 167600 [12:37:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1223588.000000 [12:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39866 MB (6% inode=99%): [12:38:51] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:38:51] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 154331 [12:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:39:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:39:50] aliasd on yarrow is CRITICAL: Connection refused [12:39:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:40:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:40:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:10] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:10] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:41:20] / on thyme is UNKNOWN: NRPE: Unable to read output [12:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:41:56] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:41:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:42:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351917 MB (6% inode=62%): [12:42:51] aliasd on mayapple is CRITICAL: No data received from host [12:43:00] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2172.000000 [12:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:43:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:02:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 598532.000000 [13:06:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [13:11:25] Hi legoktm. [13:11:28] Hi closedmouth. [13:11:32] Hi Susan. [13:11:36] yes hello [13:15:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 5837 MB (19% inode=93%): [13:15:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5828 MB (19% inode=93%): [13:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:24:00] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7219.000000 [13:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:35:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 155982.000000 [13:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739939.000000 [13:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:37:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1193906.000000 [13:37:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 437617.000000 [13:37:50] aliasd on nightshade is CRITICAL: Connection refused [13:37:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 155123 [13:37:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1227015.000000 [13:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39599 MB (6% inode=99%): [13:38:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:38:51] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 143443 [13:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:39:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:39:51] aliasd on yarrow is CRITICAL: Connection refused [13:39:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:40:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:40:09] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:10] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:40:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:41:10] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:41:10] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:41:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:41:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:41:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:41:20] / on thyme is UNKNOWN: NRPE: Unable to read output [13:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:41:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:42:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351418 MB (6% inode=62%): [13:42:51] aliasd on mayapple is CRITICAL: No data received from host [13:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:29] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:01:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:02:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 602132.000000 [14:07:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7211.000000 [14:15:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 5646 MB (18% inode=93%): [14:15:50] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5646 MB (18% inode=93%): [14:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:23:59] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10819.000000 [14:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136544.000000 [14:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743539.000000 [14:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:37:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1197506.000000 [14:37:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 441216.000000 [14:37:50] aliasd on nightshade is CRITICAL: Connection refused [14:37:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135449 [14:37:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1230464.000000 [14:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39338 MB (6% inode=99%): [14:38:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:38:50] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134640 [14:38:50] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:39:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:39:51] aliasd on yarrow is CRITICAL: Connection refused [14:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:40:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:40:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:41:11] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:41:11] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:41:11] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:41:11] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:41:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:41:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:41:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:41:21] / on thyme is UNKNOWN: NRPE: Unable to read output [14:41:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:41:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:42:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351007 MB (6% inode=62%): [14:42:51] aliasd on mayapple is CRITICAL: No data received from host [14:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:02:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 605732.000000 [15:07:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9800.000000 [15:15:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 5438 MB (18% inode=93%): [15:15:50] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5438 MB (18% inode=93%): [15:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:24:00] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14187.000000 [15:29:19] @replag [15:29:20] Betacommand: s1-rr-a: 1d 9h 57m 48s [-2.99 s/s]; s1-rr-a-wd: 6d 20h 1m 36s [+0.72 s/s]; s1-user: 10s [-]; s1-user-wd: 1w 6d 21h 3m 3s [+0.73 s/s]; s2-rr: 1m 24s [-]; s2-user: 1m 24s [-]; s2-user-c: error; s2-user-wd: 1w 1d 14h 51m 46s [+0.75 s/s] [15:29:21] Betacommand: s3-user: 25s [-0.00 s/s]; s3-user-wd: 1w 2h 34m 36s [+0.95 s/s]; s4-user-wd: 2w 6h 37m 38s [+0.96 s/s]; s5-rr-a-wd: 5d 3h 2m 58s [+0.78 s/s]; s5-user: 1d 9h 1m 6s [-3.89 s/s]; s5-user-c: 4h 32s [+0.74 s/s]; s6-user-wd: 1w 1d 25m 39s [+0.60 s/s]; s7-user-wd: 1w 37m 34s [+0.95 s/s] [15:32:04] Lydia Pintscher * [Toolserver-l] office hour on IRC [15:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119113.000000 [15:35:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744964.000000 [15:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:37:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1198909.000000 [15:37:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 442995.000000 [15:37:51] aliasd on nightshade is CRITICAL: Connection refused [15:37:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119241 [15:37:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1233952.000000 [15:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39064 MB (6% inode=99%): [15:38:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:38:50] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119762 [15:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:39:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:39:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:39:50] aliasd on yarrow is CRITICAL: Connection refused [15:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:40:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:40:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:10] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:40:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:40:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:41:10] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:41:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:41:11] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:41:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:41:20] / on thyme is UNKNOWN: NRPE: Unable to read output [15:41:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:41:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:42:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 350633 MB (6% inode=62%): [15:42:51] aliasd on mayapple is CRITICAL: No data received from host [15:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:59:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3381.000000 [16:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:02:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 608622.000000 [16:06:00] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1578.000000 [16:15:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 5291 MB (17% inode=93%): [16:15:50] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5291 MB (17% inode=93%): [16:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:24:00] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13581.000000 [16:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:11] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114501.000000 [16:35:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744934.000000 [16:36:14] @replag [16:36:15] Betacommand: s1-rr-a: 1d 8h 14m 4s [-1.55 s/s]; s1-rr-a-wd: 6d 20h 21m 13s [+0.29 s/s]; s1-user: 2m 41s [+0.04 s/s]; s1-user-wd: 1w 6d 20h 41m 34s [-0.32 s/s]; s2-user-c: error; s2-user-wd: 1w 1d 14h 55m 11s [+0.05 s/s]; s3-user: 39s [+0.00 s/s]; s3-user-wd: 1w 3h 27m 43s [+0.79 s/s] [16:36:16] Betacommand: s4-user-wd: 2w 7h 41m 9s [+0.95 s/s]; s5-rr-a-wd: 5d 3h 1m 33s [-0.02 s/s]; s5-user: 1d 7h 46m 59s [-1.11 s/s]; s5-user-c: 3h 38m 10s [-0.33 s/s]; s6-user-wd: 1w 23h 34m 25s [-0.77 s/s]; s7-user-wd: 1w 1h 31m 12s [+0.80 s/s] [16:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:37:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:37:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 442880.000000 [16:37:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1197633.000000 [16:37:50] aliasd on nightshade is CRITICAL: Connection refused [16:37:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114333 [16:37:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1237355.000000 [16:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38798 MB (6% inode=99%): [16:38:50] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 115770 [16:38:51] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:39:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:39:50] aliasd on yarrow is CRITICAL: Connection refused [16:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:40:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:10] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:40:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:40:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:40:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:41:10] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:41:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:41:10] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:41:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:41:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:41:20] / on thyme is UNKNOWN: NRPE: Unable to read output [16:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:41:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:42:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:42:51] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 350170 MB (6% inode=62%): [16:42:51] aliasd on mayapple is CRITICAL: No data received from host [16:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:43:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:01:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:02:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 611509.000000 [17:15:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 5062 MB (16% inode=93%): [17:15:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5059 MB (16% inode=93%): [17:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:24:00] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10313.000000 [17:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103518.000000 [17:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744011.000000 [17:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:37:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:37:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 442750.000000 [17:37:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1195475.000000 [17:37:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103110 [17:37:50] aliasd on nightshade is CRITICAL: Connection refused [17:37:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240836.000000 [17:38:23] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38551 MB (6% inode=99%): [17:38:50] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112775 [17:38:51] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:39:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:39:50] aliasd on yarrow is CRITICAL: Connection refused [17:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:40:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:10] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:40:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:40:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:40:51] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:41:10] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:41:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:41:10] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:41:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:41:20] / on thyme is UNKNOWN: NRPE: Unable to read output [17:41:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:41:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:42:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349813 MB (6% inode=62%): [17:42:51] aliasd on mayapple is CRITICAL: No data received from host [17:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:43:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:02:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 614337.000000 [18:16:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 4868 MB (16% inode=93%): [18:16:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4866 MB (16% inode=93%): [18:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:23:00] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3490.000000 [18:26:04] Silke Meyer * [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [18:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95644.000000 [18:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743337.000000 [18:36:00] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1685.000000 [18:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:37:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:37:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 443358.000000 [18:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1191898.000000 [18:37:50] aliasd on nightshade is CRITICAL: Connection refused [18:37:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 94957 [18:38:00] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1244349.000000 [18:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38260 MB (6% inode=99%): [18:38:50] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110235 [18:38:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:39:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:39:50] aliasd on yarrow is CRITICAL: Connection refused [18:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:40:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:11] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:40:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:40:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:40:51] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:10] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:10] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:41:20] / on thyme is UNKNOWN: NRPE: Unable to read output [18:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:41:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:42:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349454 MB (6% inode=62%): [18:42:51] aliasd on mayapple is CRITICAL: No data received from host [18:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:43:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:02:21] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 102938 MB (10% inode=99%): [19:02:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 617333.000000 [19:16:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4634 MB (15% inode=93%): [19:16:50] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4636 MB (15% inode=93%): [19:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:35:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:11] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:35:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86747.000000 [19:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743241.000000 [19:36:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:37:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:37:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 444683.000000 [19:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1190228.000000 [19:37:50] aliasd on nightshade is CRITICAL: Connection refused [19:37:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86899 [19:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37943 MB (6% inode=99%): [19:38:51] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107556 [19:38:51] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:38:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1247926.000000 [19:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:39:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:39:50] aliasd on yarrow is CRITICAL: Connection refused [19:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:40:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:40:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:10] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:11] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:41:20] / on thyme is UNKNOWN: NRPE: Unable to read output [19:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:41:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:41:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:42:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:42:50] aliasd on mayapple is CRITICAL: No data received from host [19:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349004 MB (6% inode=62%): [19:43:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:43:29] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:43:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:43:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:56:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:00:04] Marc A. Pelletier * Re: [Toolserver-l] Tool Labs maintenance (rescheduled) [20:01:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:03:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 620420.000000 [20:13:50] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2112 [20:16:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4492 MB (14% inode=93%): [20:16:50] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4486 MB (14% inode=93%): [20:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:34:10] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:34:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:35:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:35:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76018.000000 [20:35:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 746389.000000 [20:36:19] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:37:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:37:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:37:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 446560.000000 [20:37:20] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1189240.000000 [20:37:50] aliasd on nightshade is CRITICAL: Connection refused [20:37:51] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75102 [20:38:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37673 MB (6% inode=99%): [20:38:50] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 104670 [20:38:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:38:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:38:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1251459.000000 [20:39:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:39:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:39:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:39:50] aliasd on yarrow is CRITICAL: Connection refused [20:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:40:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:40:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:40:10] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:40:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:41:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:11] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:11] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:20] / on thyme is UNKNOWN: NRPE: Unable to read output [20:41:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:41:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:41:51] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:41:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:42:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:42:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348695 MB (6% inode=62%): [20:42:50] aliasd on mayapple is CRITICAL: No data received from host [20:43:09] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:43:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:45:50] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3651 [20:51:50] aliasd on nightshade is OK: TCP OK - 0.009 second response time on port 984 [500 Not found.] [20:51:51] aliasd on mayapple is OK: TCP OK - 0.006 second response time on port 984 [500 Not found.] [20:51:51] aliasd on yarrow is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [20:56:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:03:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 623693.000000 [21:07:55] aliasd on nightshade is CRITICAL: Connection refused [21:16:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 4295 MB (14% inode=93%): [21:16:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4293 MB (14% inode=93%): [21:22:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:34:25] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:25] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:35:23] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60242.000000 [21:35:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:24] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:35:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 748990.000000 [21:36:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:37:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:37:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 447743.000000 [21:37:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1184281.000000 [21:37:25] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:37:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58719 [21:38:23] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37370 MB (6% inode=99%): [21:38:55] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 100403 [21:38:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:38:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:38:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1254834.000000 [21:39:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:39:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:39:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:40:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:40:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:40:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:40:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:40:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:41:23] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:23] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:23] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:24] / on thyme is UNKNOWN: NRPE: Unable to read output [21:41:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:41:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:41:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:42:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:42:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348417 MB (6% inode=62%): [21:42:53] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3354 [21:43:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:43:24] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:43:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:56:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:01:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:04:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 626647.000000 [22:08:51] aliasd on nightshade is CRITICAL: Connection refused [22:11:51] MySQL slave on z-dat-s2-b is OK: Uptime: 141398 Threads: 10 Questions: 135418833 Slow queries: 1976 Opens: 1573733 Flush tables: 1 Open tables: 256 Queries per second avg: 957.713 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1777 [22:16:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 4152 MB (13% inode=93%): [22:16:52] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4150 MB (13% inode=93%): [22:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:29:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 16585 MB (1% inode=99%): [22:35:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:22] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:36:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:36:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44458.000000 [22:36:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:36:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 750005.000000 [22:37:12] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:37:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43930 [22:38:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 448074.000000 [22:38:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1180588.000000 [22:38:25] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:38:25] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37193 MB (6% inode=99%): [22:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:55] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 95948 [22:39:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:39:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1258152.000000 [22:40:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:40:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:40:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:40:26] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:40:26] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:40:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:40:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:40:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:40:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:41:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:41:22] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:41:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:41:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:41:23] / on thyme is UNKNOWN: NRPE: Unable to read output [22:41:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:41:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:41:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:42:22] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:42:23] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:42:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:42:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:42:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348154 MB (6% inode=62%): [22:43:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:24] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:43:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:43:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:43:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:43:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:43:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:43:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:57:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:05:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 629573.000000 [23:08:54] aliasd on nightshade is CRITICAL: Connection refused [23:16:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4086 MB (13% inode=93%): [23:16:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4086 MB (13% inode=93%): [23:23:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:35:28] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:28] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:36:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:36:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:37:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:37:23] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22306.000000 [23:37:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 748601.000000 [23:38:12] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:38:55] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21536 [23:39:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 448139.000000 [23:39:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1174164.000000 [23:39:24] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:39:24] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36958 MB (6% inode=99%): [23:39:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:40:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:40:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:40:57] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90921 [23:40:57] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:40:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:40:57] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1261359.000000 [23:41:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:23] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:23] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:25] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:26] / on thyme is UNKNOWN: NRPE: Unable to read output [23:41:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:41:31] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:41:41] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:41:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:41:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:42:22] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:42:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:42:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:42:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:43:31] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:43:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:43:31] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:43:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:43:31] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:43:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:43:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 346651 MB (6% inode=62%): [23:44:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output