[00:01:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:04:04] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:20:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294465 MB (5% inode=62%): [00:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:43:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:43:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:44:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 623883.000000 [00:44:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:44:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61921 MB (10% inode=99%): [00:44:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508698.000000 [00:44:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:45] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:44:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:44:47] aliasd on mayapple is CRITICAL: No data received from host [00:44:48] / on thyme is UNKNOWN: NRPE: Unable to read output [00:44:48] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:54] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1909214.000000 [00:45:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1142113.000000 [00:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 748224.000000 [00:45:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 938005.000000 [00:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:45:37] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1245642.000000 [00:45:38] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:56] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:57] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:46:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:46:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:46:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:47:05] Load avg. on mayapple is CRITICAL: CRITICAL - load average: 2571.51, 2566.38, 2553.08 [00:47:05] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:05] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:05] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:47:05] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:47:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:06] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:47:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:51:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:01:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:04:04] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [01:16:50] [[Special:Log/newusers]] create 10 * Wishooter * (New user account) [01:20:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294362 MB (5% inode=62%): [01:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:43:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:01] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:01] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:14] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:44:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 627483.000000 [01:44:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:44:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61856 MB (10% inode=99%): [01:44:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 504005.000000 [01:44:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:44] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:44:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:44:47] / on thyme is UNKNOWN: NRPE: Unable to read output [01:44:48] aliasd on mayapple is CRITICAL: No data received from host [01:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:44:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:54] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1912609.000000 [01:45:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 747513.000000 [01:45:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1144660.000000 [01:45:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 935954.000000 [01:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:45:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1244745.000000 [01:45:37] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:38] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:57] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:58] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:47:05] Load avg. on mayapple is CRITICAL: CRITICAL - load average: 2646.62, 2641.48, 2628.18 [01:47:05] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:47:05] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:05] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:47:05] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:06] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:47:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:47:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:47:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:47:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:51:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:01:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:04:04] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:20:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294243 MB (5% inode=62%): [02:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:43:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:43:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:14] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:14] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:44:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 631083.000000 [02:44:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:44:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61799 MB (10% inode=99%): [02:44:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 499983.000000 [02:44:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:45] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:44:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:44:47] / on thyme is UNKNOWN: NRPE: Unable to read output [02:44:48] aliasd on mayapple is CRITICAL: No data received from host [02:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1916007.000000 [02:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:36] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1145581.000000 [02:45:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 746309.000000 [02:45:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 934693.000000 [02:45:37] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:45:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1243244.000000 [02:45:38] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:57] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:58] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:46:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:48:04] Load avg. on mayapple is CRITICAL: CRITICAL - load average: 2722.67, 2717.64, 2704.35 [02:48:04] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:04] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:04] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:48:05] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:48:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:06] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:48:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:48:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:48:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:48:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:51:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:01:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:04:04] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [03:20:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:40:15] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [03:41:24] Load avg. on nightshade is WARNING: WARNING - load average: 20.94, 17.82, 11.98 [03:42:24] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:43:04] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [03:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294153 MB (5% inode=62%): [03:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:43:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:43:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:14] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:14] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:44:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:44:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61760 MB (10% inode=99%): [03:44:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 498303.000000 [03:44:45] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:44:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:44:47] / on thyme is UNKNOWN: NRPE: Unable to read output [03:44:48] aliasd on mayapple is CRITICAL: No data received from host [03:44:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:24] Load avg. on nightshade is OK: OK - load average: 8.10, 14.68, 12.25 [03:45:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 634744.000000 [03:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744932.000000 [03:45:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1148101.000000 [03:45:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 933585.000000 [03:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:45:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1242279.000000 [03:45:37] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:45:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1919390.000000 [03:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:47:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:48:04] Load avg. on mayapple is CRITICAL: CRITICAL - load average: 2798.71, 2793.71, 2780.38 [03:48:04] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:04] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:04] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:48:05] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:05] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:48:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:06] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:48:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:48:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:48:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:48:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:51:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:01:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:04:04] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:13:14] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:14] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:14] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:14] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:24] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:34] SSH on mayapple is CRITICAL: Server answer: [04:13:55] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:20:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:41:54] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:42:25] Load avg. on nightshade is WARNING: WARNING - load average: 20.29, 17.49, 12.22 [04:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294097 MB (5% inode=62%): [04:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:43:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:43:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:44:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:44:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61732 MB (10% inode=99%): [04:44:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 496900.000000 [04:44:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:45] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:44:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:44:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:44:47] / on thyme is UNKNOWN: NRPE: Unable to read output [04:44:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:14] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 638343.000000 [04:45:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744223.000000 [04:45:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1151101.000000 [04:45:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 933261.000000 [04:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:45:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1241730.000000 [04:45:37] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:45] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:55] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1922754.000000 [04:46:24] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:46:24] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:46:54] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [04:46:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:04] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [04:47:35] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:35] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:35] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:48:04] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:04] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:04] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:48:04] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:48:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:05] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:48:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:48:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:48:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:48:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:51:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:01:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:04:14] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:14] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:34] SSH on mayapple is CRITICAL: Server answer: [05:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:20:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:43:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:43:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:43:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293932 MB (5% inode=62%): [05:43:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:44:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:44:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:14] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 742713.000000 [05:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931848.000000 [05:45:36] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1152391.000000 [05:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:37] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:45:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1241808.000000 [05:45:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61679 MB (10% inode=99%): [05:45:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 495319.000000 [05:45:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [05:45:47] / on thyme is UNKNOWN: NRPE: Unable to read output [05:45:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1926048.000000 [05:46:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 642004.000000 [05:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:46:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:47:35] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:47:35] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:48:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:16] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:17] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:18] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:18] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:19] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:19] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:20] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:50:34] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 89190 MB (9% inode=99%): [05:52:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:02:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:04:14] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:44] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:21:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:04] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2140 [06:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:43:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:44:04] MySQL slave on z-dat-s2-b is OK: Uptime: 208275 Threads: 11 Questions: 212732639 Slow queries: 3448 Opens: 2091355 Flush tables: 1 Open tables: 256 Queries per second avg: 1021.402 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1584 [06:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:14] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:57] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293753 MB (5% inode=62%): [06:44:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:45:15] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:34] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931055.000000 [06:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 740816.000000 [06:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:45:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1242103.000000 [06:45:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61601 MB (10% inode=99%): [06:45:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 494351.000000 [06:45:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:44] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:45:47] / on thyme is UNKNOWN: NRPE: Unable to read output [06:45:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:54] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:54] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1155366.000000 [06:46:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1929396.000000 [06:47:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 645664.000000 [06:47:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:48:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:48:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:48:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:48:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:17] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:48:17] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:48:18] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:18] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:19] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:35] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:52:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:02:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:04:14] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:14] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:35] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:55] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:14:34] SSH on mayapple is CRITICAL: Server answer: [07:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:21:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:34] Load avg. on yarrow is CRITICAL: CRITICAL - load average: 36.50, 23.12, 11.50 [07:42:38] Load avg. on yarrow is WARNING: WARNING - load average: 16.40, 21.83, 12.56 [07:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:44:04] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1862 [07:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:44:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:55] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293276 MB (5% inode=62%): [07:44:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:45:04] MySQL slave on z-dat-s2-b is OK: Uptime: 211935 Threads: 12 Questions: 214707761 Slow queries: 3680 Opens: 2105215 Flush tables: 1 Open tables: 256 Queries per second avg: 1013.83 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1685 [07:45:15] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:45:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 738917.000000 [07:45:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 929126.000000 [07:45:37] Load avg. on yarrow is OK: OK - load average: 3.85, 13.54, 10.97 [07:45:37] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:45:38] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:45:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1242866.000000 [07:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:54] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1158476.000000 [07:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61428 MB (10% inode=99%): [07:46:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 493840.000000 [07:46:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:44] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:46:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:46:47] / on thyme is UNKNOWN: NRPE: Unable to read output [07:46:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1932663.000000 [07:47:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:27] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 649264.000000 [07:47:28] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:48:08] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:48:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:48:14] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:14] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:48:14] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:48:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:48:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:17] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:48:17] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:48:18] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:18] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:19] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:52:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:02:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:04:14] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:05:24] Load avg. on nightshade is WARNING: WARNING - load average: 9.36, 16.55, 13.47 [08:06:24] Load avg. on nightshade is OK: OK - load average: 8.04, 14.75, 13.06 [08:13:14] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:14] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:14] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:55] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:14:34] SSH on mayapple is CRITICAL: Server answer: [08:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:21:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:44:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292998 MB (5% inode=62%): [08:44:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:45:14] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:34] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 736216.000000 [08:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 927867.000000 [08:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:45:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1243829.000000 [08:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61210 MB (10% inode=99%): [08:46:44] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1161211.000000 [08:46:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 492262.000000 [08:46:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:46:47] / on thyme is UNKNOWN: NRPE: Unable to read output [08:46:48] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:54] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:47:35] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:47:35] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:48:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:17] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:18] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:18] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:19] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:19] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:20] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1936162.000000 [08:48:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:52:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:02:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:04:15] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:14:35] SSH on mayapple is CRITICAL: Server answer: [09:14:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:20:56] [[Special:Log/newusers]] create 10 * Redtigerxyz * (New user account) [09:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:44:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:44:55] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:44:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:44:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:44:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292688 MB (5% inode=62%): [09:44:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:45:15] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 734571.000000 [09:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 926116.000000 [09:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:45:54] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:45:54] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1244860.000000 [09:46:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60963 MB (9% inode=99%): [09:46:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1164419.000000 [09:46:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 491076.000000 [09:46:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:46:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:46:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:46:47] / on thyme is UNKNOWN: NRPE: Unable to read output [09:46:48] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:48:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:48:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:16] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:48:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:17] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:48:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1939595.000000 [09:48:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:48:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:48:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 656524.000000 [09:48:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:35] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:48:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:49:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:49:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:49:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:49:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:52:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:53:04] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2150 [10:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:02:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:04:15] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:14:35] SSH on mayapple is CRITICAL: Server answer: [10:14:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:19:34] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:25] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.012 second response time [10:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:44:25] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:44:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:44:55] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:44:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:44:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:44:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 283028 MB (5% inode=62%): [10:44:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:45:14] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:45:34] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 923938.000000 [10:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731785.000000 [10:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:45:55] / on willow is WARNING: DISK WARNING - free space: / 16284 MB (15% inode=98%): [10:45:55] /tmp on willow is WARNING: DISK WARNING - free space: / 16284 MB (15% inode=98%): [10:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:45:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1246654.000000 [10:46:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60718 MB (9% inode=99%): [10:46:44] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1167600.000000 [10:46:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 488162.000000 [10:46:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:46:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:46:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:46:47] / on thyme is UNKNOWN: NRPE: Unable to read output [10:46:48] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:46:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:47:35] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:48:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:48:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:48:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:16] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:48:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:17] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:18] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:48:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1942868.000000 [10:48:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:48:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:48:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 660124.000000 [10:48:35] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:49:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:49:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:54] / on willow is CRITICAL: DISK CRITICAL - free space: / 11412 MB (10% inode=98%): [10:50:54] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11412 MB (10% inode=98%): [10:51:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3623 [10:52:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:02:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:04:14] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:10:54] / on willow is OK: DISK OK - free space: / 36558 MB (34% inode=99%): [11:10:55] /tmp on willow is OK: DISK OK - free space: / 36558 MB (34% inode=99%): [11:10:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330678 MB (6% inode=62%): [11:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:15:34] SSH on mayapple is CRITICAL: Server answer: [11:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:04] Hello, CommonSense, cropbot etc. dos not work? [11:30:30] Database Error: User 'daniel_www' has exceeded the 'max_user_connections' resource (current value: 15) on sql-s4/commonswiki_p [11:32:35] Steinsplitter: Means that the tool is either too popular or not properly releasing connections [11:32:56] i think to too popular [11:33:02] -to [11:33:41] we need this tools on commons [11:33:45] :-( [11:33:46] If it's not a bug, the number of connections can be raised [11:34:20] can you do this?^^ [11:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:44:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:44:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:44:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:45:15] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 728665.000000 [11:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 923006.000000 [11:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:45:54] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1248053.000000 [11:46:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1171046.000000 [11:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60492 MB (9% inode=99%): [11:47:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 486457.000000 [11:47:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:44] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:47:47] / on thyme is UNKNOWN: NRPE: Unable to read output [11:47:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:48:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:48:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:48:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:48:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1946170.000000 [11:48:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:48:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:48:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 663724.000000 [11:48:35] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:48:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:49:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:49:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:49:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:34] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:35] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [11:51:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6669 [11:51:14] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [11:52:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:02:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:04:14] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:10:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 323469 MB (6% inode=62%): [12:13:14] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:35] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:15:34] SSH on mayapple is CRITICAL: Server answer: [12:19:03] Magnus Manske * [Toolserver-l] Migrating to labs from the toolserver [12:21:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:04] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2178 [12:28:14] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2185.000000 [12:36:14] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2127.000000 [12:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:44:14] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:44:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:44:55] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:44:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:44:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:44:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:45:15] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 722280.000000 [12:45:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 921798.000000 [12:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:45:37] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1250248.000000 [12:46:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1174663.000000 [12:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60292 MB (9% inode=99%): [12:47:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 484271.000000 [12:47:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:44] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:47:47] / on thyme is UNKNOWN: NRPE: Unable to read output [12:47:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:48:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:48:16] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:48:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:16] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:48:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1949550.000000 [12:48:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:48:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 667324.000000 [12:48:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:48:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:49:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:49:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:49:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10232 [12:53:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:56:04] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3603 [12:56:14] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [13:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:02:14] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [13:03:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:04:14] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:10:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 323151 MB (6% inode=62%): [13:11:03] rupert THURNER * Re: [Toolserver-l] Migrating to labs from the toolserver [13:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:14:03] Magnus Manske * Re: [Toolserver-l] Migrating to labs from the toolserver [13:14:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:15:15] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2205.000000 [13:15:34] SSH on mayapple is CRITICAL: Server answer: [13:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:57] I've had a task on nightshade scheduled for termination for days now :( [13:23:35] /sql on z-dat-s1-b is CRITICAL: DISK CRITICAL - free space: /sql 58172 MB (5% inode=99%): [13:28:14] s4 replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [13:30:24] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3043.000000 [13:40:24] Load avg. on nightshade is WARNING: WARNING - load average: 24.37, 17.20, 11.95 [13:43:24] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [13:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:44:25] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:44:55] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:44:56] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:44:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:44:57] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:44:57] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:44:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:45:15] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:45:24] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [13:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:34] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 922048.000000 [13:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713907.000000 [13:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:45:54] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:25] Load avg. on nightshade is WARNING: WARNING - load average: 22.84, 22.00, 15.73 [13:46:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1253624.000000 [13:46:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60048 MB (9% inode=99%): [13:47:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 480155.000000 [13:47:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:44] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:47:47] / on thyme is UNKNOWN: NRPE: Unable to read output [13:47:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:48:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1178255.000000 [13:48:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1952944.000000 [13:48:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 670923.000000 [13:48:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:49:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:49:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:49:16] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:16] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:17] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:51:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13543 [13:53:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:54:04] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2835.000000 [13:56:04] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5689 [13:57:04] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5619.000000 [14:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:03:04] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1664.000000 [14:03:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:04:15] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:10:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322824 MB (6% inode=62%): [14:11:03] Merlijn van Deen * Re: [Toolserver-l] Migrating to labs from the toolserver [14:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:15:34] SSH on mayapple is CRITICAL: Server answer: [14:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:16] @notify DaBPunkt [14:34:16] I will notify you, when I see DaBPunkt around here [14:40:03] Marc A. Pelletier * Re: [Toolserver-l] Migrating to labs from the toolserver [14:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:44:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:44:57] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:44:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:45:14] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:45:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:34] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 710795.000000 [14:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 920778.000000 [14:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1256768.000000 [14:46:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:47:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59785 MB (9% inode=99%): [14:47:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 474307.000000 [14:47:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:47:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:47:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:47:47] / on thyme is UNKNOWN: NRPE: Unable to read output [14:47:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:48:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1181121.000000 [14:48:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:48:15] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:16] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1956297.000000 [14:48:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 674524.000000 [14:48:35] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:49:14] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:14] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:49:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:49:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:16] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:52:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12675 [14:53:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:55:04] SMTP on yarrow is OK: SMTP OK - 0.229 sec. response time [14:56:04] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4637 [14:57:04] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4629.000000 [15:01:26] @replag [15:01:27] FunPika: s1-rr-a-wd: 1w 3d 19h 22m 24s [+0.46 s/s]; s1-user: 6m 46s [+0.01 s/s]; s1-user-c: 1w 19h 35m 8s [+1.00 s/s]; s1-user-wd: 2w 13h 16m 45s [+0.11 s/s]; s2-rr: 3h 1m 11s [+0.13 s/s]; s2-user: 3h 1m 11s [+0.13 s/s]; s2-user-c: error; s2-user-wd: 1w 6d 16h 14m 11s [+0.71 s/s] [15:01:28] FunPika: s3-user: 19s [-0.00 s/s]; s3-user-wd: 1w 3d 12h 58m 32s [-0.08 s/s]; s4-user-wd: 3w 1d 15h 37m 13s [+0.93 s/s]; s5-rr-a-wd: 5d 11h 20m 25s [-0.47 s/s]; s5-user: 1h 10m 36s [+0.06 s/s]; s5-user-c: 1m 51s [+0.00 s/s]; s6-user-wd: 1w 3d 15h 40m 48s [-0.28 s/s]; s7-user-wd: 1w 1d 5h 13m 43s [-0.64 s/s] [15:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:02:18] [[Mail]] ! 10https://wiki.toolserver.org/w/index.php?diff=7987&oldid=5465&rcid=21869 * Tim.Landscheidt * (+488) (Document how to restart a stuck mail queue for operators.) [15:03:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:04:15] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:06:05] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3380.000000 [15:06:05] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3371 [15:10:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322410 MB (6% inode=62%): [15:13:05] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1693.000000 [15:13:05] MySQL slave on z-dat-s5-b is OK: Uptime: 238755 Threads: 4 Questions: 472672306 Slow queries: 2905 Opens: 162163 Flush tables: 1 Open tables: 256 Queries per second avg: 1979.737 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1680 [15:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:15:34] SSH on mayapple is CRITICAL: Server answer: [15:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:04] / on wolfsbane is WARNING: DISK WARNING - free space: / 6225 MB (20% inode=93%): [15:37:04] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6225 MB (20% inode=93%): [15:42:34] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 95532 MB (9% inode=99%): [15:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:43:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:44:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:44:57] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:44:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:45:15] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:45:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:45:34] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 704705.000000 [15:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 919775.000000 [15:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:45:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1259189.000000 [15:46:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:47:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:47:35] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:48:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1183683.000000 [15:48:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:48:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:48:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:16] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:48:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1959635.000000 [15:48:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:48:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59514 MB (9% inode=99%): [15:48:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 467207.000000 [15:48:44] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:48:44] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:48:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:48:47] / on thyme is UNKNOWN: NRPE: Unable to read output [15:48:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:49:14] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:14] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:49:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:49:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:16] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:24] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 678183.000000 [15:49:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:52:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7608 [15:53:25] Load avg. on nightshade is WARNING: WARNING - load average: 12.89, 15.22, 11.44 [15:53:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:54:24] Load avg. on nightshade is OK: OK - load average: 12.75, 14.80, 11.53 [15:54:36] Deleting files on /home takes ages ... [16:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:04:24] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:04:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:10:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322050 MB (6% inode=62%): [16:12:29] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [16:12:29] Load avg. on nightshade is WARNING: WARNING - load average: 22.94, 19.53, 14.68 [16:12:55] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [16:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:14:04] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:14:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:15:34] SSH on mayapple is CRITICAL: Server answer: [16:20:15] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [16:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:14] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:04] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3530 [16:28:15] aliasd on nightshade is OK: TCP OK - 0.075 second response time on port 984 [500 Not found.] [16:35:04] MySQL slave on z-dat-s2-b is OK: Uptime: 243736 Threads: 13 Questions: 243511279 Slow queries: 6888 Opens: 2291165 Flush tables: 1 Open tables: 256 Queries per second avg: 999.78 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1721 [16:38:04] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5982 MB (19% inode=93%): [16:38:04] / on wolfsbane is WARNING: DISK WARNING - free space: / 5982 MB (19% inode=93%): [16:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:44:25] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:44:25] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:44:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:44:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:45:24] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 700310.000000 [16:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 917895.000000 [16:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:45:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:03] rupert THURNER * [Toolserver-l] Fwd: Your message to Labs-l awaits moderator approval [16:46:25] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1261806.000000 [16:46:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:54] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:47:35] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:47:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:45] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1185682.000000 [16:48:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:48:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:16] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59260 MB (9% inode=99%): [16:48:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 463200.000000 [16:48:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:47] / on thyme is UNKNOWN: NRPE: Unable to read output [16:48:47] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:48] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 22.83, 24.30, 23.56 [16:48:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:49:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:49:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:49:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:25] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:49:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1963024.000000 [16:49:25] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:49:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:49:26] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:27] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:27] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:28] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 681789.000000 [16:49:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:54:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:04:24] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:04:24] Load avg. on nightshade is WARNING: WARNING - load average: 9.03, 15.20, 19.38 [17:04:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:09:03] Huib Laurens * Re: [Toolserver-l] Fwd: Your message to Labs-l awaits moderator approval [17:10:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321682 MB (6% inode=62%): [17:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:15:04] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:15:34] SSH on mayapple is CRITICAL: Server answer: [17:15:55] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:21:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:04] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5748 MB (19% inode=93%): [17:38:04] / on wolfsbane is WARNING: DISK WARNING - free space: / 5748 MB (19% inode=93%): [17:40:24] Load avg. on nightshade is WARNING: WARNING - load average: 9.25, 16.54, 19.65 [17:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:43:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:44:54] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:44:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:44:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:44:56] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:44:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:45:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:45:24] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:45:25] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 697653.000000 [17:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 914351.000000 [17:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:45:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:45:45] Load avg. on nightshade is WARNING: WARNING - load average: 15.08, 12.92, 16.88 [17:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264543.000000 [17:46:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:54] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:47:35] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:47:35] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1187471.000000 [17:48:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:48:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:16] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:16] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:16] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58973 MB (9% inode=99%): [17:48:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 461388.000000 [17:48:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:48:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:48:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:45] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:46] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:47] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:48:48] / on thyme is UNKNOWN: NRPE: Unable to read output [17:48:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:49:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:49:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:25] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1966229.000000 [17:49:25] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:25] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:26] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:27] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:27] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:28] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:52:24] / on ortelius is WARNING: DISK WARNING - free space: / 5984 MB (19% inode=92%): [17:54:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:04:24] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:04:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:10:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321344 MB (6% inode=62%): [18:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:13:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:14:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:14:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:15:04] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:15:34] SSH on mayapple is CRITICAL: Server answer: [18:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:04] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4248 MB (14% inode=93%): [18:39:05] / on wolfsbane is WARNING: DISK WARNING - free space: / 4248 MB (14% inode=93%): [18:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:44:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:44:57] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:44:57] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:44:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:45:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:45:24] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:45:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 694903.000000 [18:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 909593.000000 [18:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:45:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1267288.000000 [18:46:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:54] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:47:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:47:35] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:47:35] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:48:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1189374.000000 [18:48:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58723 MB (9% inode=99%): [18:48:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 460520.000000 [18:48:44] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:48:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:48:46] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:47] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:48] / on thyme is UNKNOWN: NRPE: Unable to read output [18:49:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:49:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:49:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:49:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:49:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:49:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:49:45] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:49:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:50:24] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1969689.000000 [18:50:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:24] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:26] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:27] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:27] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:28] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:28] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 689050.000000 [18:52:24] / on ortelius is WARNING: DISK WARNING - free space: / 4953 MB (16% inode=92%): [18:53:04] Kolossos * Re: [Toolserver-l] Migrating to labs from the toolserver [18:55:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:04:24] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:04:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:10:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321060 MB (6% inode=62%): [19:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:13:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:14:14] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:14:15] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:14:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:15:04] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:15:34] SSH on mayapple is CRITICAL: Server answer: [19:16:03] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3133 MB (10% inode=93%): [19:16:04] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3133 MB (10% inode=93%): [19:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:58] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:58] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:58] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:44:58] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:58] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:58] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:58] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:58] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:58] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:45:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:45:24] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:45:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:34] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 693395.000000 [19:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 905760.000000 [19:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:45:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:45:54] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:45:54] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1270288.000000 [19:46:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:47:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:48:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1191637.000000 [19:48:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58332 MB (9% inode=99%): [19:48:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 459691.000000 [19:48:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:46] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:46] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:47] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:48:47] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:48:48] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:48] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:49] / on thyme is UNKNOWN: NRPE: Unable to read output [19:49:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:49:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:49:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:14] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:14] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:49:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:49:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:24] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:49:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:49:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:49:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:50:28] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1973082.000000 [19:50:28] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:28] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:28] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:28] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:29] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 692651.000000 [19:51:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:52:24] / on ortelius is WARNING: DISK WARNING - free space: / 4032 MB (13% inode=92%): [19:55:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:04:24] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:04:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:10:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320640 MB (6% inode=62%): [20:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:14:14] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:14:14] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:14:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:14:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:15:04] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:15:34] SSH on mayapple is CRITICAL: Server answer: [20:16:03] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2841 MB (9% inode=93%): [20:16:04] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2841 MB (9% inode=93%): [20:21:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:44:24] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:55] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:56] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 690750.000000 [20:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 903503.000000 [20:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:45:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:45:54] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:54] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:24] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:46:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1273335.000000 [20:46:45] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:47:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:48:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1192359.000000 [20:48:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:48:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58075 MB (9% inode=99%): [20:48:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 458425.000000 [20:48:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:48:47] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:48:47] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:48] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:48] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:49] / on thyme is UNKNOWN: NRPE: Unable to read output [20:49:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:49:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:49:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:49:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:49:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:49:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:49:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:50:25] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:50:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1976463.000000 [20:50:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:25] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:50:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:50:26] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:50:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 696251.000000 [20:51:14] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:24] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:51:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:24] / on ortelius is WARNING: DISK WARNING - free space: / 3423 MB (11% inode=92%): [20:54:54] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [20:56:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:00:24] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 2997 MB (10% inode=92%): [21:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:04:25] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:04:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:10:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320305 MB (6% inode=62%): [21:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:14:14] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:14:14] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:14:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:15:04] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:15:34] SSH on mayapple is CRITICAL: Server answer: [21:16:04] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2125 MB (7% inode=93%): [21:16:04] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2125 MB (7% inode=93%): [21:21:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:44:24] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:56] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:56] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:34] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 687100.000000 [21:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 900344.000000 [21:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:45:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:45:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:24] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1276238.000000 [21:46:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:47:35] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:47:35] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:48:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1192918.000000 [21:48:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 455281.000000 [21:48:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57765 MB (9% inode=99%): [21:48:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:48:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:48:46] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:47] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:47] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:48] / on thyme is UNKNOWN: NRPE: Unable to read output [21:49:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:49:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:49:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:49:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:49:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:49:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:50:25] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:50:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:25] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:50:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1979928.000000 [21:50:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:50:26] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:50:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 699851.000000 [21:51:14] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:24] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:51:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:54:54] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [21:56:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:00:24] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 1146 MB (3% inode=92%): [22:02:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:04:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:05:24] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:10:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319920 MB (6% inode=62%): [22:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:13:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:14:14] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:14:15] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:14:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:15:04] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:15:35] SSH on mayapple is CRITICAL: Server answer: [22:16:04] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 996 MB (3% inode=93%): [22:16:04] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 996 MB (3% inode=93%): [22:21:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:44:14] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:44:24] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:44:55] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:55] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 682619.000000 [22:45:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 897370.000000 [22:45:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:56] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:46:24] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:34] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:46:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1279364.000000 [22:46:45] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:34] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:47:35] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:48:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1192735.000000 [22:48:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 452509.000000 [22:48:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57600 MB (9% inode=99%): [22:48:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:48:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:48:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:48:46] / on thyme is UNKNOWN: NRPE: Unable to read output [22:48:47] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:48:47] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:48:48] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:48:55] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:49:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:49:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:49:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:15] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:49:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:49:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:49:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:49:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:49:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:50:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:25] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:50:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1983399.000000 [22:50:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:25] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:50:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:50:26] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:50:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 703451.000000 [22:51:14] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:52:24] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:54:54] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [22:57:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:00:24] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 386 MB (1% inode=92%): [23:02:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:04:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:05:24] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:10:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319536 MB (5% inode=62%): [23:13:14] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:14] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:14] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:14:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:15:05] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:15:35] SSH on mayapple is CRITICAL: Server answer: [23:16:04] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 106 MB (0% inode=93%): [23:16:04] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 106 MB (0% inode=93%): [23:21:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:44:24] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:44:54] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:55] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 679521.000000 [23:45:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 895672.000000 [23:45:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:36] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:36] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:55] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:56] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:04] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:46:24] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:34] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:46:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1282071.000000 [23:46:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:46] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:05] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:05] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:05] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:05] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:35] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:47:36] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:48:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1191407.000000 [23:48:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 450643.000000 [23:48:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57365 MB (9% inode=99%): [23:48:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:45] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:48:46] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:48:46] / on thyme is UNKNOWN: NRPE: Unable to read output [23:48:47] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:47] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:48] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:49:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:49:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:49:24] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:49:24] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:24] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:49:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:49:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:49:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:50:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:26] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:50:27] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:50:27] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:27] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1986835.000000 [23:50:27] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:50:27] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:50:27] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 707052.000000 [23:51:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:25] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:54:55] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [23:57:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output