[00:01:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12803.000000 [00:03:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:03:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:03:52] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:03:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:04:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319366 MB (5% inode=63%): [00:04:42] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:05:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:05:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 797717.000000 [00:05:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:05:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1229108.000000 [00:05:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 362331.000000 [00:05:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 992076.000000 [00:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:05:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:05:41] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:01] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:11] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:12] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:23] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:06:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:42] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:07:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:07:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:07:22] / on thyme is UNKNOWN: NRPE: Unable to read output [00:07:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:07:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:07:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:07:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:07:24] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [00:07:24] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:07:25] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:07:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1667628.000000 [00:07:26] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:07:26] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:07:27] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 618704.000000 [00:12:52] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3383.000000 [00:17:11] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105965.000000 [00:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62813 MB (10% inode=99%): [00:17:23] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324216 MB (6% inode=63%): [00:20:52] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1381.000000 [00:26:42] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3503.000000 [00:28:22] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5968 MB (19% inode=93%): [00:28:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5967 MB (19% inode=93%): [00:31:42] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1533.000000 [00:32:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:33:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:33:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:34:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:36:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:36:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:36:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:36:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:36:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:01] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:52] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:04:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:05:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:05:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 798368.000000 [01:05:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 992797.000000 [01:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:02] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 365990.000000 [01:06:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1227902.000000 [01:06:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:22] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:06:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:42] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:07:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:07:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:08:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 618397.000000 [01:08:22] / on thyme is UNKNOWN: NRPE: Unable to read output [01:08:23] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:08:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:08:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:08:23] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:08:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [01:08:23] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [01:08:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:08:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:08:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1671015.000000 [01:08:25] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:26] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:26] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:17:11] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109565.000000 [01:17:13] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62591 MB (10% inode=99%): [01:17:23] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322115 MB (6% inode=63%): [01:28:22] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5830 MB (19% inode=93%): [01:28:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5830 MB (19% inode=93%): [01:32:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:33:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:34:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:36:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:36:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:36:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:36:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:36:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:51] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:04:42] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:05:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:05:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 799076.000000 [02:05:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:05:31] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 993502.000000 [02:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:05:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:05:41] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:01] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 369591.000000 [02:06:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1225371.000000 [02:06:12] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:22] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:06:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:42] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:07:01] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:07:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:08:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 617824.000000 [02:08:22] / on thyme is UNKNOWN: NRPE: Unable to read output [02:08:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:08:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:08:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:08:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [02:09:22] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [02:09:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [02:09:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:09:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:09:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1674335.000000 [02:09:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:09:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:09:24] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60443 MB (9% inode=99%): [02:17:12] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113165.000000 [02:17:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321248 MB (6% inode=63%): [02:28:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5701 MB (19% inode=93%): [02:28:22] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5701 MB (19% inode=93%): [02:32:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:34:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:34:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:36:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:36:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:36:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:36:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:36:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:04] MZMcBride * [Toolserver-l] Toolserver --> Labs [03:03:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:03:51] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:52] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:42] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:05:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:05:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 799472.000000 [03:05:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:05:31] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 993127.000000 [03:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:05:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:05:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:05:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:02] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 373191.000000 [03:06:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1223498.000000 [03:06:12] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:22] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:22] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:06:22] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:42] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:07:01] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:07:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:07:23] /tmp on wolfsbane is OK: DISK OK - free space: / 6406 MB (21% inode=93%): [03:07:23] / on wolfsbane is OK: DISK OK - free space: / 6406 MB (21% inode=93%): [03:08:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616870.000000 [03:08:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:08:22] / on thyme is UNKNOWN: NRPE: Unable to read output [03:08:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:08:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:08:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [03:09:22] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [03:09:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [03:09:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:09:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1677642.000000 [03:09:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:09:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:10:22] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:10:22] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:12:04] Jeremy Baron * Re: [Toolserver-l] Toolserver --> Labs [03:14:04] Marc A. Pelletier * Re: [Toolserver-l] Toolserver --> Labs [03:17:12] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116765.000000 [03:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62268 MB (10% inode=99%): [03:17:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321795 MB (6% inode=63%): [03:21:54] [[Special:Log/newusers]] create 10 * Sora * (New user account) [03:32:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:34:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:36:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:36:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:36:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:36:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:36:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:01:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 6251 MB (20% inode=93%): [04:01:23] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6251 MB (20% inode=93%): [04:03:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:03:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:03:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:03:51] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:03:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:42] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:05:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:05:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 800053.000000 [04:05:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:05:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:05:41] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 994041.000000 [04:05:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:01] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 376791.000000 [04:06:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1221864.000000 [04:06:12] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:22] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:22] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:22] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:06:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:42] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:07:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:07:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:08:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 615935.000000 [04:08:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:08:23] / on thyme is UNKNOWN: NRPE: Unable to read output [04:08:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:08:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:08:23] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [04:09:22] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [04:09:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [04:09:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:09:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1680836.000000 [04:09:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:09:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:11:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:11:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:12] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120365.000000 [04:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62064 MB (10% inode=99%): [04:17:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321183 MB (6% inode=63%): [04:28:02] Free Memory on damiana is WARNING: WARNING - 5.3% (447520 kB) free! [04:30:16] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7948&oldid=7927&rcid=21822 * 185.15.59.202 * (+0) (updated page) [04:32:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:34:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (409396 kB) free! [04:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:34:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:35:02] Free Memory on damiana is WARNING: WARNING - 5.5% (457268 kB) free! [04:36:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:36:33] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:36:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:36:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:36:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:01:23] / on wolfsbane is WARNING: DISK WARNING - free space: / 6132 MB (20% inode=93%): [05:02:22] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6129 MB (20% inode=93%): [05:03:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:03:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:03:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:03:52] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:03:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:04:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:05:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:05:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 800075.000000 [05:05:13] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:05:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:05:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:05:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:01] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:06:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:13] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 380391.000000 [05:06:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1220209.000000 [05:06:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:22] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:23] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:06:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 994403.000000 [05:06:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:06:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:42] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:07:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:07:02] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (129816 kB) free! [05:07:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:07:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:07:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:08:13] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 612909.000000 [05:08:23] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:09:22] / on thyme is UNKNOWN: NRPE: Unable to read output [05:09:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:09:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:09:23] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [05:09:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [05:09:23] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:09:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:09:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1684145.000000 [05:09:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:09:25] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:11:26] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:11:26] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:12] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123965.000000 [05:17:13] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61897 MB (10% inode=99%): [05:17:23] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320500 MB (6% inode=63%): [05:25:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:25:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:25:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:26:22] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:29:59] eth0 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:30:57] FC 0/23 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:31:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:31:07] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:31:07] FC 0/16 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:31:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:31:08] Sun Grid Engine execd on wolfsbane is CRITICAL: (Service Check Timed Out) [05:31:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:31:11] toolserver is throwing 404's and 500's [05:32:38] /tmp on wolfsbane is CRITICAL: (Service Check Timed Out) [05:32:38] / on wolfsbane is CRITICAL: (Service Check Timed Out) [05:48:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:48:11] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:11] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:11] / on thyme is UNKNOWN: NRPE: Unable to read output [05:48:20] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:20] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:20] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:20] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:21] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:40] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [05:49:30] Free Memory on damiana is OK: OK - 92.4% (7740128 kB) free. [05:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:54:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:54:23] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320205 MB (6% inode=63%): [05:54:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:54:23] / on wolfsbane is WARNING: DISK WARNING - free space: / 5993 MB (20% inode=93%): [05:54:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:54:32] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5994 MB (20% inode=93%): [05:54:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:56:04] Ryan Lane * Re: [Toolserver-l] Toolserver --> Labs [05:56:42] aliasd on yarrow is CRITICAL: Connection refused [06:03:52] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2184.000000 [06:03:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2180.000000 [06:06:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 802442.000000 [06:06:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:06:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:06:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:06:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:06:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:06:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 384030.000000 [06:07:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 996442.000000 [06:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1221435.000000 [06:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:07:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:07:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:09:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 614359.000000 [06:09:32] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:09:32] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:09:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1687326.000000 [06:10:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:10:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:17:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61722 MB (10% inode=99%): [06:18:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127607.000000 [06:18:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:27:51] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [06:27:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [06:47:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:42] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:42] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:42] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:47:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:48:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:48:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:13] / on thyme is UNKNOWN: NRPE: Unable to read output [06:48:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:54:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:54:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:54:23] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320016 MB (6% inode=63%): [06:54:23] / on wolfsbane is WARNING: DISK WARNING - free space: / 5765 MB (19% inode=93%): [06:54:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:55:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:55:35] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5772 MB (19% inode=93%): [06:57:41] aliasd on yarrow is CRITICAL: Connection refused [07:06:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 806041.000000 [07:06:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:06:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:06:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 387629.000000 [07:07:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1000042.000000 [07:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1225035.000000 [07:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:07:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:11] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:11] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:07:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:41] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:07:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:09:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 617959.000000 [07:10:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:10:21] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:10:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [07:10:32] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1690645.000000 [07:17:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61543 MB (10% inode=99%): [07:18:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:19:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131267.000000 [07:27:51] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7224.000000 [07:27:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7219.000000 [07:47:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:42] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:42] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:42] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:47:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:48:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:48:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:12] / on thyme is UNKNOWN: NRPE: Unable to read output [07:48:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:49:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:49:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:49:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:49:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:49:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:49:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:54:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:54:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:55:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319648 MB (6% inode=63%): [07:55:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5522 MB (18% inode=93%): [07:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:55:31] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5530 MB (18% inode=93%): [07:57:41] aliasd on yarrow is CRITICAL: Connection refused [08:06:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:06:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 391229.000000 [08:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1003642.000000 [08:07:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 809701.000000 [08:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1228635.000000 [08:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:07:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:07:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:07:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:41] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:08:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 621559.000000 [08:10:21] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [08:10:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [08:10:32] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [08:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1694111.000000 [08:11:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:17:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61330 MB (10% inode=99%): [08:18:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:19:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134867.000000 [08:21:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319540 MB (5% inode=63%): [08:27:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10824.000000 [08:27:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10819.000000 [08:47:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:01] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:48:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:48:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:12] / on thyme is UNKNOWN: NRPE: Unable to read output [08:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:42] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:42] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:42] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:48:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:50:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:50:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:50:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:54:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:54:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:55:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5257 MB (17% inode=93%): [08:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:56:31] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5258 MB (17% inode=93%): [08:57:41] aliasd on yarrow is CRITICAL: Connection refused [09:06:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 394830.000000 [09:07:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1007242.000000 [09:07:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 813302.000000 [09:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1232235.000000 [09:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:07:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:11] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:07:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:08:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:08:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:08:41] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:09:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 625159.000000 [09:09:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [09:10:32] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [09:10:32] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [09:10:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1697545.000000 [09:11:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61051 MB (10% inode=99%): [09:18:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:19:02] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138467.000000 [09:21:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319244 MB (5% inode=63%): [09:28:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14484.000000 [09:28:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14479.000000 [09:37:31] /tmp on wolfsbane is OK: DISK OK - free space: / 11954 MB (39% inode=93%): [09:38:22] / on wolfsbane is OK: DISK OK - free space: / 12580 MB (41% inode=93%): [09:44:52] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3202.000000 [09:47:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:51] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1203.000000 [09:48:01] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:48:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:48:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:12] / on thyme is UNKNOWN: NRPE: Unable to read output [09:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:41] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:51] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:51] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:49:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:49:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:49:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:49:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:50:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:54:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:54:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:55:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:56:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3367.000000 [09:58:42] aliasd on yarrow is CRITICAL: Connection refused [10:00:51] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1575.000000 [10:02:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320090 MB (6% inode=63%): [10:06:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 398430.000000 [10:07:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1008550.000000 [10:07:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 814722.000000 [10:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1231946.000000 [10:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:07:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:07:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:07:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:07:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:08:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:09:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 624267.000000 [10:09:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:09:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:21] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [10:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1700988.000000 [10:11:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [10:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [10:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [10:17:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60760 MB (9% inode=99%): [10:18:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:19:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 142067.000000 [10:47:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:01] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:41] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:51] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:49:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:12] / on thyme is UNKNOWN: NRPE: Unable to read output [10:49:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:50:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:54:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:54:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:58:41] aliasd on yarrow is CRITICAL: Connection refused [11:02:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319644 MB (6% inode=63%): [11:06:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 402030.000000 [11:07:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 815147.000000 [11:07:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1008586.000000 [11:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1228587.000000 [11:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:07:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:07:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:08:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:08:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:09:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:09:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 619499.000000 [11:10:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [11:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1704451.000000 [11:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [11:11:32] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [11:12:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60497 MB (9% inode=99%): [11:18:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145727.000000 [11:26:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319524 MB (5% inode=63%): [11:47:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:41] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:51] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:49:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:12] / on thyme is UNKNOWN: NRPE: Unable to read output [11:49:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:50:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:50:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:50:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:50:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:54:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:54:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:58:41] aliasd on yarrow is CRITICAL: Connection refused [12:06:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 405629.000000 [12:06:53] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 815155.000000 [12:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1224786.000000 [12:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:07:21] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1010926.000000 [12:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:07:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:07:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:08:13] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:13] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:09:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:09:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:10:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616600.000000 [12:10:21] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [12:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1707940.000000 [12:10:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [12:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [12:12:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:17:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60173 MB (9% inode=99%): [12:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 149327.000000 [12:26:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319195 MB (5% inode=63%): [12:47:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:48:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:48:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:41] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:51] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:51] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:49:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:13] / on thyme is UNKNOWN: NRPE: Unable to read output [12:49:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:21] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:54:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:54:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:58:04] DaB. * Re: [Toolserver-l] Toolserver --> Labs [12:58:42] aliasd on yarrow is CRITICAL: Connection refused [13:06:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 409230.000000 [13:07:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 814454.000000 [13:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1222202.000000 [13:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:07:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:07:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:00] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1011440.000000 [13:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:09:04] Marc A. Pelletier * Re: [Toolserver-l] Toolserver --> Labs [13:09:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:10:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:10:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [13:10:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 613556.000000 [13:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [13:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [13:11:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1711520.000000 [13:12:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:17:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59917 MB (9% inode=99%): [13:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 152927.000000 [13:26:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318916 MB (5% inode=63%): [13:47:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:48:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:49:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:12] / on thyme is UNKNOWN: NRPE: Unable to read output [13:49:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:41] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:54:23] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:54:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:55:21] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:55:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:58:41] aliasd on yarrow is CRITICAL: Connection refused [14:06:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 412830.000000 [14:07:01] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 813484.000000 [14:07:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1219656.000000 [14:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:07:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:08:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1012741.000000 [14:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:09:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:09:13] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:13] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:09:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:10:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:10:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [14:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [14:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [14:11:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1715054.000000 [14:11:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 610453.000000 [14:12:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59670 MB (9% inode=99%): [14:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156527.000000 [14:26:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318513 MB (5% inode=62%): [14:47:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:48:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:48:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:49:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:12] / on thyme is UNKNOWN: NRPE: Unable to read output [14:49:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:41] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:55:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:55:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:58:42] aliasd on yarrow is CRITICAL: Connection refused [15:06:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 416430.000000 [15:06:53] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1217371.000000 [15:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:07:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:07:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:08:00] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 812374.000000 [15:08:01] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:09:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1013443.000000 [15:09:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:09:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:09:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:10:21] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [15:11:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [15:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [15:11:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1718569.000000 [15:12:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 608120.000000 [15:12:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:12:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59406 MB (9% inode=99%): [15:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 160127.000000 [15:26:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318007 MB (5% inode=62%): [15:47:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:48:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:48:51] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:51] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:49:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:12] / on thyme is UNKNOWN: NRPE: Unable to read output [15:49:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:42] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:55:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:55:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:58:42] aliasd on yarrow is CRITICAL: Connection refused [16:06:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 420030.000000 [16:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1216340.000000 [16:07:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:07:02] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:08:01] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 811330.000000 [16:08:01] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:08:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:08:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:09:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:09:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:09:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:10:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1014233.000000 [16:10:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [16:11:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [16:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [16:11:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1722015.000000 [16:12:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 606860.000000 [16:12:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:12:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59203 MB (9% inode=99%): [16:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 163727.000000 [16:26:21] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317568 MB (5% inode=62%): [16:47:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:48:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:51] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:49:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:13] / on thyme is UNKNOWN: NRPE: Unable to read output [16:49:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:42] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:51:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:54:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:55:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:56:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:58:42] aliasd on yarrow is CRITICAL: Connection refused [17:06:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 423630.000000 [17:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1213740.000000 [17:07:12] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:07:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:11:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 810219.000000 [17:11:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:11:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:11:20] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:20] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:11:22] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:11:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1016549.000000 [17:11:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:11:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:11:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [17:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [17:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [17:11:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1725462.000000 [17:12:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 605544.000000 [17:12:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:12:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:13:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58910 MB (9% inode=99%): [17:18:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 167328.000000 [17:26:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317129 MB (5% inode=62%): [17:47:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:02] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:48:02] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 97981 MB (10% inode=99%): [17:48:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:01] aliasd on yarrow is CRITICAL: Connection refused [18:07:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1213501.000000 [18:07:11] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:07:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:07:29] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:07:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:51] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:51] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 427290.000000 [18:08:00] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 809162.000000 [18:08:01] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:08:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:09:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:10:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1017703.000000 [18:10:11] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:10:11] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:10:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [18:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [18:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [18:11:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1728900.000000 [18:12:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 602432.000000 [18:12:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:12:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:13:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58613 MB (9% inode=99%): [18:18:01] Free Memory on turnera is WARNING: WARNING - 5.7% (476952 kB) free! [18:18:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:20:00] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 170928.000000 [18:20:01] Free Memory on turnera is OK: OK - 7.1% (595724 kB) free. [18:26:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 316210 MB (5% inode=62%): [18:29:02] Free Memory on turnera is WARNING: WARNING - 6.9% (580756 kB) free! [18:33:02] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.612 second response time [18:34:00] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.576 second response time [18:35:51] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.117 second response time [18:47:30] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:13] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:49:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:49:11] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:11] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:12] / on thyme is UNKNOWN: NRPE: Unable to read output [18:50:21] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:41] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:21] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:54:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:55:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:55:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:55:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:56:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:59:41] aliasd on yarrow is CRITICAL: Connection refused [19:07:01] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1211138.000000 [19:07:11] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:07:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:07:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:51] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:51] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 430890.000000 [19:08:11] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:08:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:09:01] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 808221.000000 [19:09:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:10:00] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1020862.000000 [19:10:11] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:11] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:10:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:10:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [19:11:31] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [19:11:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [19:11:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1732377.000000 [19:12:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 600783.000000 [19:12:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:12:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:13:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58324 MB (9% inode=99%): [19:18:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 174528.000000 [19:27:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315866 MB (5% inode=62%): [19:47:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:12] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:49:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:49:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:11] / on thyme is UNKNOWN: NRPE: Unable to read output [19:50:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:41] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:51] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:54:03] DaB. * [Toolserver-announce] Survey: Moving to Labs [19:54:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:55:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:55:21] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:55:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:55:59] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7950&oldid=7949&rcid=21824 * 134.84.121.69 * (+48) (/* Survey */ +me ) [19:56:16] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7951&oldid=7950&rcid=21825 * 134.84.121.69 * (+2) (/* Survey */ + next ) [19:56:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:59:00] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7952&oldid=7951&rcid=21826 * 78.50.77.140 * (+57) (+euku) [19:59:29] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7953&oldid=7952&rcid=21827 * Addshore * (+36) () [19:59:42] aliasd on yarrow is CRITICAL: Connection refused [20:00:15] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7954&oldid=7953&rcid=21828 * Addshore * (+24) (/* Survey */ add some) [20:03:34] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7955&oldid=7954&rcid=21829 * MF-Warburg * (+33) (/* Survey */ ) [20:04:04] Liangent * Re: [Toolserver-l] Survey: Moving to Labs [20:05:04] Aaron Halfaker * Re: [Toolserver-l] Survey: Moving to Labs [20:06:04] Tim Landscheidt * Re: [Toolserver-l] Survey: Moving to Labs [20:07:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1208112.000000 [20:07:12] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:08:00] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 434490.000000 [20:08:11] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:08:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:08:42] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:42] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:08:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:51] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:01] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 807401.000000 [20:09:39] [[Labs-Moving-Survey]] 10https://wiki.toolserver.org/w/index.php?diff=7956&oldid=7955&rcid=21830 * Siebrand * (+23) (/* Survey */ ) [20:09:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:10:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1023139.000000 [20:10:06] [[Labs-Moving-Survey]] 10https://wiki.toolserver.org/w/index.php?diff=7957&oldid=7956&rcid=21831 * Siebrand * (+18) (/* Survey */ ) [20:10:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:10:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:04] seth * Re: [Toolserver-l] Survey: Moving to Labs [20:11:21] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [20:11:30] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [20:11:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1735779.000000 [20:12:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 599978.000000 [20:12:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:12:31] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [20:13:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:14:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:04] Liangent * Re: [Toolserver-l] Survey: Moving to Labs [20:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58098 MB (9% inode=99%): [20:18:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:19:04] Ryan Lane * Re: [Toolserver-l] Survey: Moving to Labs [20:20:00] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 178128.000000 [20:23:35] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7958&oldid=7957&rcid=21832 * Addshore * (+119) (/* Survey */ lets get a few more details? :)) [20:24:02] Free Memory on turnera is WARNING: WARNING - 6.0% (505508 kB) free! [20:24:11] [[User:Addshore]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7959&rcid=21833 * Addshore * (+35) ({{softredirect|meta:User:Addshore}}) [20:24:20] [[User talk:Addshore]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7960&rcid=21834 * Addshore * (+40) ({{softredirect|meta:User talk:Addshore}}) [20:27:21] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315421 MB (5% inode=62%): [20:29:01] Free Memory on turnera is OK: OK - 7.2% (600552 kB) free. [20:43:56] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7961&oldid=7958&rcid=21835 * Svick * (+17) (added myseld) [20:47:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:47:41] aliasd on yarrow is OK: TCP OK - 0.008 second response time on port 984 [500 Not found.] [20:48:11] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:49:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:49:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:11] / on thyme is UNKNOWN: NRPE: Unable to read output [20:50:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:22] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:42] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:22] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:22] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:22] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:30] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:52:28] @replag [20:52:29] koopal: s1-rr-a: 2d 2h 1m 24s [-]; s1-rr-a-wd: 1w 3d 1h 4m 37s [-]; s1-user-c: 5d 1h 26m 10s [-]; s1-user-wd: 1w 6d 23h 11m 7s [-]; s2-rr: 2d 2h 1m 43s [-]; s2-user: 2d 2h 1m 43s [-]; s2-user-c: error; s2-user-wd: 1w 4d 20h 14m 51s [-] [20:52:30] koopal: s3-user-wd: 1w 3d 5h 12m 53s [-]; s4-user-wd: 2w 6d 2h 49m 1s [-]; s5-rr-a-wd: 6d 22h 34m 33s [-]; s5-user: 2d 2h 1m 28s [-]; s6-user-wd: 1w 3d 15h 49m 55s [-]; s7-user-wd: 1w 2d 8h 1m 50s [-] [20:52:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:55:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:55:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:56:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:56:21] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:57:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:58:00] Free Memory on turnera is WARNING: WARNING - 6.9% (576748 kB) free! [20:59:00] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (413636 kB) free! [20:59:21] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [20:59:21] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [20:59:38] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7962&oldid=7961&rcid=21836 * Alexxw * (+91) (/* Survey */ ) [20:59:50] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [21:00:00] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7963&oldid=7962&rcid=21837 * Alexxw * (+0) () [21:00:20] MySQL on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [21:07:29] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:08:00] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1206911.000000 [21:08:11] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:08:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:08:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 438150.000000 [21:09:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:09:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:09:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:09:50] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:10:00] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 806167.000000 [21:10:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:11:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [21:11:11] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:11:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:11:22] MySQL slave on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [21:11:31] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [21:11:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1739235.000000 [21:12:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 599440.000000 [21:12:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:12:30] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [21:14:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:14:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57822 MB (9% inode=99%): [21:19:22] MySQL on z-dat-s5-b is OK: Uptime: 1531 Threads: 1 Questions: 38 Slow queries: 0 Opens: 29 Flush tables: 1 Open tables: 21 Queries per second avg: 0.24 [21:19:51] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1572.000000 [21:20:01] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181692.000000 [21:27:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315105 MB (5% inode=62%): [21:29:51] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2172.000000 [21:31:56] [[Labs-Moving-Survey]] 10https://wiki.toolserver.org/w/index.php?diff=7964&oldid=7963&rcid=21838 * Stwalkerster * (+142) (+acc) [21:38:31] ugh [21:42:04] Platonides * Re: [Toolserver-l] Survey: Moving to Labs [21:43:53] @replag [21:43:54] FunPika: s1-rr-a: error; s1-rr-a-wd: error; s1-user-c: 5d 2h 17m 35s [-]; s1-user-wd: 1w 6d 23h 38m 16s [-]; s2-rr: error; s2-user: error; s2-user-c: error; s2-user-wd: error [21:43:55] FunPika: s3-user: 12s [-]; s3-user-wd: 1w 3d 5h 29m 7s [-]; s4-user-wd: 2w 6d 3h 36m 54s [-]; s5-rr-a-wd: 6d 22h 36m 11s [-]; s5-user: 2d 2h 36m 7s [-]; s5-user-c: 50m 18s [-]; s6-user-wd: 1w 3d 15h 53m 10s [-]; s7-user-wd: 1w 2d 7h 58m 38s [-] [21:48:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:48:28] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:59] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:49:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:50:28] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:59] / on thyme is UNKNOWN: NRPE: Unable to read output [21:51:09] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:39] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:39] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:39] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:39] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:48] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:49] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:49] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:18] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:18] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:18] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:19] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:28] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:53:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:53:58] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [21:55:48] MySQL on z-dat-s2-b is OK: Uptime: 3777 Threads: 7 Questions: 4556 Slow queries: 1 Opens: 186 Flush tables: 1 Open tables: 177 Queries per second avg: 1.206 [21:56:08] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:56:19] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:57:09] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:58:29] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:59:59] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [22:08:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1209737.000000 [22:08:29] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:08:59] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 441753.000000 [22:09:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:09:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:09:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:09:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:10:08] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:09] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:10:29] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:59] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 807931.000000 [22:11:18] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:11:18] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:11:28] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:11:48] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1742585.000000 [22:11:58] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 181941 [22:11:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2201.000000 [22:11:59] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 181125 [22:11:59] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1028052.000000 [22:12:09] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:12:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:13:08] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [22:13:08] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 601513.000000 [22:14:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:15:49] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57585 MB (9% inode=99%): [22:20:59] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180175.000000 [22:27:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 314515 MB (5% inode=62%): [22:30:54] [[Special:Log/newusers]] create 10 * Danilo * (New user account) [22:35:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3642.000000 [22:37:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:37:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:48:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:48:28] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:08] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:59] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:49:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:50:28] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:59] / on thyme is UNKNOWN: NRPE: Unable to read output [22:51:09] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:19] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:19] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:38] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:39] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:39] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:39] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:48] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:48] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:18] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:18] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:18] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:18] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:28] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:53:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:53:59] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7215.000000 [22:54:59] MySQL on z-dat-s1-b is OK: Uptime: 7320 Threads: 6 Questions: 2211 Slow queries: 0 Opens: 75 Flush tables: 1 Open tables: 66 Queries per second avg: 0.302 [22:56:08] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:56:19] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:57:09] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:58:29] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:08:28] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:08:59] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 445353.000000 [23:08:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1213396.000000 [23:09:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:09:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:09:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:09:58] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:10:08] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:10:28] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:59] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 811530.000000 [23:11:18] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:11:18] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:11:28] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:11:48] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1745916.000000 [23:11:58] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 175603 [23:11:58] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 161565 [23:11:58] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1031652.000000 [23:12:08] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:12:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:13:09] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 187091 [23:13:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 605112.000000 [23:14:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:15:48] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:04] Hazard-SJ * Re: [Toolserver-l] enwiki_p corruption [23:18:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57310 MB (9% inode=99%): [23:20:58] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 159485.000000 [23:27:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 313036 MB (5% inode=62%): [23:35:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7241.000000 [23:37:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:37:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:48:29] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3355.000000 [23:49:08] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:59] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:49:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:50:28] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:58] / on thyme is UNKNOWN: NRPE: Unable to read output [23:51:08] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:18] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:19] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:39] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:39] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:39] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:39] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:49] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:49] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:59] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1485.000000 [23:52:18] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:18] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:18] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:19] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:28] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:53:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:53:59] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8447.000000 [23:54:52] [[Labs-Moving-Survey]] 10https://wiki.toolserver.org/w/index.php?diff=7965&oldid=7964&rcid=21840 * APPER * (+176) () [23:56:08] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:56:18] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:57:09] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:58:29] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output