[00:00:33] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:00:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 257037 MB (4% inode=62%): [00:02:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2102590.000000 [00:04:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1269729.000000 [00:04:54] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [00:04:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [00:04:55] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:04:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:04:55] aliasd on nightshade is CRITICAL: Connection refused [00:04:55] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:04:56] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:05:12] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:05:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 365795.000000 [00:05:13] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:21] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:33] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:05:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48264 MB (7% inode=99%): [00:05:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 585215.000000 [00:05:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:05:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:05:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:55] SSH on mayapple is CRITICAL: Server answer: [00:05:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:06:03] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:06:03] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:06:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:03] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:06:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:12] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:06:13] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:13] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:13] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:06:13] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:06:13] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:06:14] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:14] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:06:15] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:06:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:06:32] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:06:49] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:06:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:07:02] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:07:13] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:07:13] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:07:13] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:07:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:07:13] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:07:13] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:07:13] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:07:13] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:07:14] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:07:14] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:07:15] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:07:16] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:07:32] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:32] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:33] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:33] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:42] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:07:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:03] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:03] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:03] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:12] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:08:12] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:08:13] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:13] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:08:13] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 813688.000000 [00:08:32] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:08:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:08:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:08:56] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:08:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:08:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:09:32] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:09:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:09:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:12:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:16:32] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21658.000000 [00:24:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:25:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:25:32] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:28:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:54:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:55:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1358061.000000 [00:55:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:58:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 883945.000000 [01:00:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:00:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 256899 MB (4% inode=62%): [01:02:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2105668.000000 [01:04:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1270245.000000 [01:04:55] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [01:04:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [01:04:55] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:04:55] aliasd on nightshade is CRITICAL: Connection refused [01:04:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:04:55] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [01:05:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:05:13] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:05:13] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 365391.000000 [01:05:13] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48040 MB (7% inode=99%): [01:05:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 580179.000000 [01:05:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:05:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:05:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:55] SSH on mayapple is CRITICAL: Server answer: [01:05:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:06:03] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:06:03] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:06:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:03] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:06:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:13] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:13] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:06:13] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:13] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:06:14] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:06:14] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:06:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:06:15] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:06:16] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:06:43] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:06:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:03] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:07:13] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:07:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:13] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:13] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:07:13] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:13] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:07:14] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:07:14] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:07:15] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:15] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:17] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:17] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:07:18] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:07:32] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:32] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:32] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:33] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:34] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:42] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:07:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:02] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:03] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:03] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:12] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:08:12] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:08:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:12] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:08:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:13] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 812465.000000 [01:08:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:08:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:08:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:08:55] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:55] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:08:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:08:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:09:32] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:09:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:09:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:12:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:16:32] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25258.000000 [01:24:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:25:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:26:32] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:28:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:36:35] yarrow seems to have frozen. ssh times out, a session of mine there hangs. nightshade seems okay. [01:37:21] Correction: ssh works, but *slowly* ... (still waiting for a prompt) [01:38:03] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:38:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:41:03] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:41:09] alkamid has a process running with 1.6 GByte RES ... [01:47:55] Free Memory on turnera is WARNING: WARNING - 5.8% (487532 kB) free! [01:51:55] Free Memory on turnera is CRITICAL: CRITICAL - 4.1% (346404 kB) free! [01:52:55] Free Memory on turnera is WARNING: WARNING - 6.3% (525384 kB) free! [01:54:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:59:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 887605.000000 [02:00:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 256783 MB (4% inode=62%): [02:02:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2108802.000000 [02:04:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:04:59] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [02:04:59] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:04:59] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [02:04:59] aliasd on nightshade is CRITICAL: Connection refused [02:05:00] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [02:05:00] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1271748.000000 [02:05:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:39] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [02:05:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 579359.000000 [02:05:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:05:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:06:09] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:06:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:09] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:06:09] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 365278.000000 [02:06:10] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45841 MB (7% inode=99%): [02:06:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:06:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:06:49] SSH on mayapple is CRITICAL: Server answer: [02:06:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:06:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:07:08] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:07:08] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:07:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:29] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:07:39] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:07:39] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:07:39] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:07:39] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:07:39] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:07:40] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:07:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:07:41] / on thyme is UNKNOWN: NRPE: Unable to read output [02:07:49] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:07:49] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:07:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:07:59] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:07:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:08:09] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:08:09] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:08:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 812359.000000 [02:08:18] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:08:38] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:08:38] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:08:59] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:08:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:09:00] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:09:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:09:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:09:19] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:09:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:09:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:10:09] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:10:18] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:13:28] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:16:59] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28881.000000 [02:25:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:25:49] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:26:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:30:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:36:18] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:59] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:54:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:55:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1358693.000000 [02:55:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:56:18] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:56:18] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:56:19] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:56:19] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:56:19] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:56:19] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:56:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:56:59] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:56:59] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:57:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:57:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:57:09] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:57:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:57:09] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:57:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:57:09] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:57:10] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:57:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:57:11] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:57:11] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:57:12] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:57:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:59:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 891205.000000 [03:01:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 254535 MB (4% inode=62%): [03:02:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2111921.000000 [03:04:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:04:59] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [03:04:59] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [03:04:59] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:04:59] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [03:05:00] aliasd on nightshade is CRITICAL: Connection refused [03:05:00] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1272606.000000 [03:05:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:05:39] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [03:05:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 580208.000000 [03:05:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:05:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:05:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:06:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:09] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:06:09] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:06:09] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367005.000000 [03:06:10] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:30] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48140 MB (7% inode=99%): [03:06:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:06:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:06:49] SSH on mayapple is CRITICAL: Server answer: [03:06:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:06:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:07:08] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:07:08] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:07:08] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:07:28] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:07:39] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:07:39] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:07:39] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:07:39] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:07:39] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:07:39] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:07:39] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:07:40] / on thyme is UNKNOWN: NRPE: Unable to read output [03:07:49] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:07:49] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:07:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:07:59] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:07:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:08:09] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:08:09] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:08:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 813796.000000 [03:08:28] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:08:38] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:08:38] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:08:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:08:59] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:08:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:09:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:09:18] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:09:19] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:09:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:09:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:10:09] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:09] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:10:18] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:13:29] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:16:59] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32481.000000 [03:25:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:25:49] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:26:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:30:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:35:18] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:35:59] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:40:18] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [03:41:19] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [03:42:19] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:42:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:54:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:55:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1358711.000000 [03:55:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:56:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:56:59] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:56:59] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:57:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:57:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:57:09] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:57:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:57:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:57:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:57:10] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:57:11] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:57:11] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:57:12] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:57:12] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:57:13] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:57:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:59:48] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 894805.000000 [04:00:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 254470 MB (4% inode=62%): [04:02:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2115028.000000 [04:04:24] [[Special:Log/newusers]] create 10 * Brawogekdoa23 * (New user account) [04:04:59] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [04:04:59] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [04:04:59] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:04:59] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [04:04:59] aliasd on nightshade is CRITICAL: Connection refused [04:05:05] [[User talk:Brawogekdoa23]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7995&rcid=21883 * Brawogekdoa23 * (+2212) (Created page with "'''Những điều cần biết về Kỳ thi NAT-TEST''' Kỳ thi NAT-TEST là một kỳ thi đo năng lực tiếng Nhật của bạn giống như 日本語能力試験 Niho...") [04:05:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:39] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [04:05:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 578970.000000 [04:05:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:05:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:05:59] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1272113.000000 [04:06:09] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:06:09] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:06:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:09] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367803.000000 [04:06:10] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:29] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47459 MB (7% inode=99%): [04:06:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:06:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:06:49] SSH on mayapple is CRITICAL: Server answer: [04:06:58] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.047 second response time [04:06:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:07:00] [[Special:Log/block]] block 10 * Legoktm * (blocked [[02User:Brawogekdoa2310]] with an expiry time of infinite (account creation disabled): spambot) [04:07:08] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:07:08] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:07:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:15] [[Special:Log/delete]] delete 10 * Legoktm * (deleted "[[02User talk:Brawogekdoa2310]]": spam) [04:07:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:29] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:07:39] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:39] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:39] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:39] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:39] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:40] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:07:41] / on thyme is UNKNOWN: NRPE: Unable to read output [04:07:49] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:07:49] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:07:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:07:59] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:07:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:08:09] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:08:09] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:08:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 812735.000000 [04:08:28] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:08:38] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:08:38] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:08:58] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:08:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:09:00] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:09:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:09:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:09:19] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:09:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:09:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:10:08] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:10:18] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:13:28] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:16:59] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36081.000000 [04:25:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:25:49] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:26:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:30:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:34:19] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:34:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:35:59] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:37:18] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:54:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:55:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1359898.000000 [04:55:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:56:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:56:58] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:56:59] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:09] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:09] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:09] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:09] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:10] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:11] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:19] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:19] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:19] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:20] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:20] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:57:21] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:58:08] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:58:09] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:58:09] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:58:09] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:58:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:58:09] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:58:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:58:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:59:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 898405.000000 [05:00:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 254327 MB (4% inode=62%): [05:02:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2118269.000000 [05:04:59] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [05:04:59] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:04:59] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [05:04:59] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [05:04:59] aliasd on nightshade is CRITICAL: Connection refused [05:05:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:05:39] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:05:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 577106.000000 [05:05:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:05:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:05:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:05:59] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1271336.000000 [05:06:09] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:06:09] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:06:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368118.000000 [05:06:09] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:06:10] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:06:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:06:29] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47297 MB (7% inode=99%): [05:06:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:06:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:06:49] SSH on mayapple is CRITICAL: Server answer: [05:06:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:06:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:07:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:07:09] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:07:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:07:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:07:28] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:07:39] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:07:39] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:07:39] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:07:39] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:07:39] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:07:39] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:07:40] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:07:40] / on thyme is UNKNOWN: NRPE: Unable to read output [05:07:49] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:07:49] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:07:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:08:01] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:08:01] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:08:09] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:08:09] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:08:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 812240.000000 [05:08:29] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:08:38] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:08:38] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:08:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:09:00] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:09:00] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:09:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:09:18] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:09:19] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:09:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:09:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:10:09] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:10:18] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:13:29] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:16:59] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39682.000000 [05:25:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:25:49] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:26:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:30:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:35:58] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:39:18] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:39:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:54:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:55:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1359732.000000 [05:55:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:56:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:56:58] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:56:59] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:57:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:57:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:57:09] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:09] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:10] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:10] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:10] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:57:11] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:57:19] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:57:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:19] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:57:19] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:19] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:57:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:58:08] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:58:09] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:58:09] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:58:09] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:58:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:58:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:58:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:58:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:58:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:58:18] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:59:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 902004.000000 [06:00:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 254200 MB (4% inode=62%): [06:02:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2121267.000000 [06:04:59] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [06:04:59] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [06:04:59] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:04:59] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:05:00] aliasd on nightshade is CRITICAL: Connection refused [06:05:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:39] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:05:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 573916.000000 [06:05:58] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:05:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:06:00] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1270587.000000 [06:06:08] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:06:09] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:06:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:06:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368169.000000 [06:06:09] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:09] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:09] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:29] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47125 MB (7% inode=99%): [06:06:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:06:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:06:49] SSH on mayapple is CRITICAL: Server answer: [06:06:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:06:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:07:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:07:09] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:07:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:28] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:07:38] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:07:38] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:07:39] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:07:39] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:07:39] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:07:39] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:07:39] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:07:40] / on thyme is UNKNOWN: NRPE: Unable to read output [06:07:48] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:07:49] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:07:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:08:08] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:08:09] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:08:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 810996.000000 [06:08:29] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:08:39] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:08:39] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:08:59] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:08:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:09:18] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:09:18] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:09:28] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:09:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:09:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:09:59] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:09:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:10:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:10:09] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:10:19] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:13:28] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:16:58] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43282.000000 [06:17:59] Free Memory on damiana is WARNING: WARNING - 6.3% (528464 kB) free! [06:19:49] Free Memory on damiana is OK: OK - 7.3% (614768 kB) free. [06:25:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:25:49] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:26:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:27:49] Free Memory on damiana is CRITICAL: CRITICAL - 3.2% (264808 kB) free! [06:30:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:35:59] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:39:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:54:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:55:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1360436.000000 [06:55:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:56:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:56:59] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:56:59] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:57:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:57:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:57:10] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:10] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:10] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:57:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:57:11] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:12] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:12] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:19] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:19] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:57:19] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:57:19] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:57:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:58:08] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:58:08] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:09] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:58:18] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:18] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:19] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:58:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:19] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:58:19] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:59:48] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 905605.000000 [07:01:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 254043 MB (4% inode=62%): [07:02:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2124157.000000 [07:04:59] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:04:59] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [07:05:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:05:39] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 570541.000000 [07:05:49] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 87081 MB (8% inode=99%): [07:05:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:59] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [07:05:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:05:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:05:59] aliasd on nightshade is CRITICAL: Connection refused [07:06:00] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:06:00] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1269921.000000 [07:06:09] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:06:09] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:06:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:06:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368158.000000 [07:06:09] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:10] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46929 MB (7% inode=99%): [07:06:48] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:06:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:06:49] SSH on mayapple is CRITICAL: Server answer: [07:06:58] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:06:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:07:09] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:07:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:07:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:29] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:07:39] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:39] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:39] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:39] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:39] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:40] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:40] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:41] / on thyme is UNKNOWN: NRPE: Unable to read output [07:07:49] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:07:49] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:07:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:08:09] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:08:09] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:08:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 809993.000000 [07:08:28] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:08:38] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:08:39] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:08:59] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:08:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:19] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:09:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:59] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:10:08] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:10:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:10:18] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:13:28] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:16:59] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46882.000000 [07:25:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:26:49] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:26:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:30:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:35:59] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:39:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:54:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:55:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1361801.000000 [07:56:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:56:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:56:58] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:56:59] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:57:08] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:57:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:57:18] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:57:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:57:19] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:57:19] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:57:19] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:57:19] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:57:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:58:09] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:58:09] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:58:09] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:58:09] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:58:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:10] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:58:11] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:58:11] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:58:12] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:58:18] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:18] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:18] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:18] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:19] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:19] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:59:08] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:59:09] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:59:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 909204.000000 [08:01:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 253843 MB (4% inode=62%): [08:02:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2127173.000000 [08:04:59] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:04:59] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [08:05:38] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [08:05:38] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:05:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 568617.000000 [08:05:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:59] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [08:05:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:06:08] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:06:08] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:06:08] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:06:08] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367515.000000 [08:06:09] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:09] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:09] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46736 MB (7% inode=99%): [08:06:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:06:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:06:49] SSH on mayapple is CRITICAL: Server answer: [08:06:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:06:59] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [08:06:59] aliasd on nightshade is CRITICAL: Connection refused [08:06:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:06:59] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1269834.000000 [08:07:00] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:07:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:07:09] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:07:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:28] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:07:38] / on thyme is UNKNOWN: NRPE: Unable to read output [08:07:38] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:38] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:39] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:39] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:39] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:39] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:40] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:07:49] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:07:49] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:07:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:08:08] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:08:09] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:08:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 808330.000000 [08:08:29] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:08:39] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:08:39] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:08:59] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:08:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:18] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:19] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:09:28] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:58] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:10:00] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:10:09] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:10:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:10:19] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:10:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:13:38] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:14:55] petrb@yarrow:~$ acctrenew [08:14:57] -bash: acctrenew: command not found [08:15:05] anything that can be done to fix that? [08:15:21] there is nothing wrong in my PATH [08:15:34] it just seems that a command is missing [08:16:58] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50482.000000 [08:25:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:26:49] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:26:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:30:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:35:59] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:36:48] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [08:36:49] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [08:37:08] MySQL on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [08:37:19] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:37:59] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:39:19] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:39:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:52:59] aliasd on yarrow is CRITICAL: Connection refused [08:54:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:55:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1362098.000000 [08:56:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:56:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:56:59] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:56:59] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:57:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:57:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:57:18] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:57:18] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:57:18] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:57:19] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:57:19] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:57:19] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:57:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:58:09] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:58:09] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:58:09] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:58:09] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:58:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:10] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:58:10] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:58:11] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:11] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:58:12] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:58:19] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:19] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:19] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:19] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:09] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:59:09] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:18] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:59:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 912805.000000 [09:01:59] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:01:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 253658 MB (4% inode=62%): [09:03:00] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2130235.000000 [09:04:58] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:04:59] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [09:05:38] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:39] MySQL slave on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [09:05:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:05:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 567403.000000 [09:05:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:59] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [09:05:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:06:08] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:06:09] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:06:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:06:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 366769.000000 [09:06:09] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:09] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46506 MB (7% inode=99%): [09:06:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:06:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:06:49] SSH on mayapple is CRITICAL: Server answer: [09:07:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:07:00] aliasd on nightshade is CRITICAL: Connection refused [09:07:00] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [09:07:00] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:07:00] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [09:07:00] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:07:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:07:09] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:07:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:28] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:07:38] / on thyme is UNKNOWN: NRPE: Unable to read output [09:07:39] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:39] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:39] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:39] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:39] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:40] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:49] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:07:49] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:07:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:07:49] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (381140 kB) free! [09:08:08] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:08:08] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:08:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 805042.000000 [09:08:29] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:08:39] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:08:39] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:08:59] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:08:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:19] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:09:28] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:59] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:10:08] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:10:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:10:19] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:10:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:13:39] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:17:00] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [09:21:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:22:19] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:25:20] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:25:28] FC 0/14 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:25:28] FC 0/20 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:25:29] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:25:29] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:25:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:26:19] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:26:22] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:26:22] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:22] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:26:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:29] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:26:29] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:26:49] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:26:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:27:20] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:27:26] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:26] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:27:26] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:27:26] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:27:26] FC 0/22 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:27:27] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:27:28] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:27:28] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:27:29] FC 0/13 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:27:29] FC 0/9 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:20] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:28:30] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:28:31] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:31] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:31] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:31] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:28:32] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:32] FC 0/21 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:33] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:33] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:28:34] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:34] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:35] FC 0/16 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:28:35] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:29:14] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:29:23] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:29:23] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:29:23] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:29:24] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:29:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:29:24] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:29:24] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:29:24] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:29:25] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:29:25] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: No answer from host [09:29:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:29:32] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: No answer from host [09:29:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: No answer from host [09:30:24] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:36] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:37] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:37] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:37] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:30:37] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:38] FC 0/22 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:38] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:38] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:39] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:39] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:40] FC 0/9 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:30:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:30:41] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: No answer from host [09:31:00] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:31:31] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:31:44] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:31:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:31:45] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:31:45] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:31:45] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: No answer from host [09:32:30] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:32:37] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:32:38] FC 0/21 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:32:38] FC 0/10 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [09:34:18] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:34:33] Load avg. on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:34:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:34:34] /tmp on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:34:34] /mnt on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:34:35] / on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:34:36] RAID on thyme is CRITICAL: (Service Check Timed Out) [09:34:37] /sql on z-dat-s1-b is CRITICAL: (Service Check Timed Out) [09:34:38] APT on z-dat-s1-b is CRITICAL: (Service Check Timed Out) [09:34:38] APT on sage is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:35:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:35:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:36:48] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:47:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:47:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:47:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367300.000000 [09:47:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:53:04] aliasd on yarrow is CRITICAL: Connection refused [09:53:04] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:55:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:10:04] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2217.000000 [10:11:04] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2178.000000 [10:15:04] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2197.000000 [10:18:44] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:04] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2209.000000 [10:29:33] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.014 second response time [10:34:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3658.000000 [10:35:06] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3617.000000 [10:36:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:37:34] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:38:11] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3357.000000 [10:38:43] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3637.000000 [10:39:35] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:41:06] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1373.000000 [10:43:42] [[Special:Log/newusers]] create 10 * TT22 * (New user account) [10:44:06] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000 [10:45:06] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3353.000000 [10:46:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:46:53] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:47:05] aliasd on nightshade is CRITICAL: Connection refused [10:47:05] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:47:05] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [10:47:05] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3416.000000 [10:47:06] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:47:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:47:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2136099.000000 [10:47:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [10:47:08] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [10:47:08] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [10:47:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 919240.000000 [10:49:33] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.010 second response time [10:50:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3594.000000 [10:52:06] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2089 [10:52:07] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3510 [10:52:07] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2942 [10:52:07] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1769.000000 [10:52:15] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3245 [10:53:05] MySQL slave on z-dat-s7-a is OK: Uptime: 143136 Threads: 9 Questions: 143006469 Slow queries: 1984 Opens: 333470 Flush tables: 1 Open tables: 3240 Queries per second avg: 999.95 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1620 [10:53:06] aliasd on yarrow is CRITICAL: Connection refused [10:55:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:57:07] MySQL slave on z-dat-s3-a is OK: Uptime: 144200 Threads: 32 Questions: 233853323 Slow queries: 4524 Opens: 1674996 Flush tables: 1 Open tables: 16383 Queries per second avg: 1621.729 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1651 [10:59:06] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:03:06] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1714.000000 [11:03:15] MySQL slave on rosemary is OK: Uptime: 126493 Threads: 37 Questions: 120917976 Slow queries: 24200 Opens: 12012 Flush tables: 1 Open tables: 2929 Queries per second avg: 955.926 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1665 [11:10:06] MySQL slave on z-dat-s1-b is OK: Uptime: 483419 Threads: 22 Questions: 418852562 Slow queries: 9466 Opens: 2351 Flush tables: 1 Open tables: 476 Queries per second avg: 866.437 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1742 [11:17:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:19:44] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:35] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.006 second response time [11:36:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:38:06] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:40:24] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [11:42:33] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:44:24] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [11:46:53] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:47:05] aliasd on nightshade is CRITICAL: Connection refused [11:47:05] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:47:05] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [11:47:05] SSH on mayapple is CRITICAL: Server answer: [11:47:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 569719.000000 [11:47:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:47:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1366571.000000 [11:47:06] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 803279.000000 [11:47:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:47:24] / on thyme is UNKNOWN: NRPE: Unable to read output [11:47:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:47:24] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:47:24] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:24] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:24] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:25] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:47:26] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [11:47:26] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [11:47:27] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 252898 MB (4% inode=62%): [11:47:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:38] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:39] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:39] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:44] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:53] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:54] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:54] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:48:04] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:48:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:48:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2139302.000000 [11:48:06] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [11:48:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [11:48:06] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [11:48:06] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 922900.000000 [11:48:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:48:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:48:07] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:48:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:48:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45906 MB (7% inode=99%): [11:48:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 369130.000000 [11:48:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:15] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:53:34] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:55:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:13:07] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:34:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:35:07] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:37:33] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:54] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:47:06] aliasd on nightshade is CRITICAL: Connection refused [12:47:06] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:47:06] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [12:47:06] SSH on mayapple is CRITICAL: Server answer: [12:47:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 568276.000000 [12:47:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:47:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1367641.000000 [12:47:08] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 798738.000000 [12:47:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:16] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:16] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:16] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:47:17] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:47:25] / on thyme is UNKNOWN: NRPE: Unable to read output [12:47:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:47:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:47:25] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:26] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:26] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:27] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:27] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 252487 MB (4% inode=62%): [12:47:28] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:47:28] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [12:47:29] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [12:47:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:41] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:41] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:43] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:43] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:43] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:43] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:53] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:53] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:53] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:48:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:48:06] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:48:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2142261.000000 [12:48:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [12:48:07] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 926500.000000 [12:48:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [12:48:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [12:48:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:48:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:48:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:48:16] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:48:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45679 MB (7% inode=99%): [12:48:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367832.000000 [12:48:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:48:16] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:16] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:12] aliasd on yarrow is CRITICAL: Connection refused [12:55:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:59:06] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:37:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:38:24] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:39:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:40:24] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [13:42:25] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [13:46:53] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:47:06] aliasd on nightshade is CRITICAL: Connection refused [13:47:06] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:47:06] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [13:47:07] SSH on mayapple is CRITICAL: Server answer: [13:47:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:47:07] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 795691.000000 [13:47:08] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 567503.000000 [13:47:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1368172.000000 [13:47:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:47:25] / on thyme is UNKNOWN: NRPE: Unable to read output [13:47:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:47:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:47:26] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:26] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:26] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:27] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:27] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 252144 MB (4% inode=62%): [13:47:28] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:47:28] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [13:47:29] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [13:47:29] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:47:30] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [13:47:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:44] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:54] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:54] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:54] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:48:06] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:48:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:48:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2145212.000000 [13:48:06] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [13:48:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [13:48:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [13:48:07] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 930100.000000 [13:48:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:48:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:48:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:48:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:16] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:16] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:48:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45412 MB (7% inode=99%): [13:48:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367265.000000 [13:54:06] aliasd on yarrow is CRITICAL: Connection refused [13:55:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:59:06] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [14:17:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:39:21] @replag [14:44:31] tsbot doesn't seem to have come back to life :-). [14:46:53] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:07] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:47:07] aliasd on nightshade is CRITICAL: Connection refused [14:47:07] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [14:47:07] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [14:47:07] SSH on mayapple is CRITICAL: Server answer: [14:47:07] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 794431.000000 [14:47:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 565874.000000 [14:47:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:47:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1368398.000000 [14:47:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:47:25] / on thyme is UNKNOWN: NRPE: Unable to read output [14:47:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:47:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:34] MySQL slave on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [14:47:34] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:47:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:44] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:53] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:54] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:54] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:48:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:48:07] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [14:48:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2148434.000000 [14:48:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [14:48:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [14:48:08] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [14:48:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 933701.000000 [14:48:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:48:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:48:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:48:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:16] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:16] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:48:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367614.000000 [14:49:34] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:34] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:49:34] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:49:34] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:49:34] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:49:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:52:39] like everything else around here [14:53:29] When I restart aliasd on nightshade, it dies within seconds. There seems to be a bug in there, but it's hard to debug this way. Well, in a few days we should have another root. [14:54:06] aliasd on nightshade is OK: TCP OK - 0.010 second response time on port 984 [500 Not found.] [14:54:07] aliasd on yarrow is CRITICAL: Connection refused [14:55:06] aliasd on yarrow is OK: TCP OK - 0.008 second response time on port 984 [500 Not found.] [14:55:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:00:06] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:35:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:36:07] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:41:34] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:41:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:42:15] aliasd on yarrow is OK: TCP OK - 9.893 second response time on port 984 [200 wp@dabpunkt.eu] [15:46:54] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:47:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:47:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:47:25] / on thyme is UNKNOWN: NRPE: Unable to read output [15:47:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:47:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:34] MySQL slave on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [15:47:34] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:47:43] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:43] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:53] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:53] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:53] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:06] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:48:06] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [15:48:06] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:48:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:48:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2151747.000000 [15:48:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [15:48:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [15:48:08] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [15:48:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 937300.000000 [15:48:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:48:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:48:10] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 563544.000000 [15:48:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1368967.000000 [15:48:11] SSH on mayapple is CRITICAL: Server answer: [15:48:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:48:25] MySQL on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [15:48:25] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 251325 MB (4% inode=62%): [15:48:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:25] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:26] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:48:26] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [15:48:34] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:48:34] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:21] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:49:21] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:49:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367560.000000 [15:49:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44923 MB (7% inode=99%): [15:49:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:49:21] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:01:06] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:24:43] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:33] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.009 second response time [16:46:54] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:34] MySQL slave on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [16:47:34] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:34] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:43] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:44] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:53] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:54] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:54] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:48:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:48:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:48:24] / on thyme is UNKNOWN: NRPE: Unable to read output [16:48:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:48:34] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:34] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:05] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:49:05] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [16:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:49:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:49:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2154257.000000 [16:49:06] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [16:49:06] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [16:49:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [16:49:07] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 940960.000000 [16:49:08] SSH on mayapple is CRITICAL: Server answer: [16:49:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:49:09] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560249.000000 [16:49:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1369732.000000 [16:49:10] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 791537.000000 [16:49:24] MySQL on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [16:49:24] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:49:25] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 250788 MB (4% inode=62%): [16:49:25] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:26] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:49:26] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [16:49:27] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [16:49:27] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:49:28] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [16:49:28] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:49:29] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:50:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:50:15] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:50:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44662 MB (7% inode=99%): [16:50:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367878.000000 [16:50:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:52:34] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.009 second response time [16:57:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:02:07] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:17:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:30:12] Betacommand: Can you tell if my toolserver account has been approved or not? [17:30:14] https://jira.toolserver.org/browse/ACCAPP-644 [17:33:11] greenrosetta: doesnt look like it was created yet [17:33:23] over a month [17:33:24] :( [17:36:24] greenrosetta: looks like it was less than a week ago [17:36:33] the original request [17:36:49] greenrosetta: things take time [17:41:43] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:43:05] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:46:38] Betacommand: are you on the wikimedia listserv? [17:46:53] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:34] MySQL slave on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [17:47:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:44] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:54] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:54] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:54] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:54] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:48:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:48:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:48:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:48:25] / on thyme is UNKNOWN: NRPE: Unable to read output [17:48:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:48:34] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:34] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:34] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:05] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:49:06] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [17:49:07] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:49:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:49:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2157213.000000 [17:49:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [17:49:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [17:49:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [17:49:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 944560.000000 [17:49:08] SSH on mayapple is CRITICAL: Server answer: [17:49:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:49:09] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 791282.000000 [17:49:10] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556571.000000 [17:49:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1370751.000000 [17:49:25] MySQL on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [17:49:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:49:25] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 250541 MB (4% inode=62%): [17:49:26] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:26] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:27] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:49:27] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [17:49:28] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [17:49:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:49:29] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [17:49:44] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:50:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:16] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:50:16] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:50:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44406 MB (7% inode=99%): [17:50:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368378.000000 [17:50:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:50:17] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:16] Yes [17:51:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:34] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.022 second response time [17:57:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:05:07] Free Memory on turnera is WARNING: WARNING - 5.7% (477564 kB) free! [18:11:07] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (374220 kB) free! [18:12:06] Free Memory on turnera is OK: OK - 8.3% (699236 kB) free. [18:12:45] Betacommand: Would you mind trying out my stub application on tools? I'd like to see if someone else can get it to work quickly [18:26:53] greenrosetta: Participants in a JIRA issue don't get mailed (anymore) when the issue is changed such as when you added your data, so it's probably easiest to remind DaBPunkt directly. [18:27:15] how would I do that? [18:27:24] He doens't appear to have contact info anywhere [18:30:01] Either here on IRC or per mail. He sends mail to toolserver-l from his personal account, but you could of course also try ts-admins@toolserver.org. [18:36:03] Hello all [18:37:10] hello [18:37:14] maintenance tonight [18:37:31] will reboot ortelius in a minute [18:38:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:39:07] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:40:34] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:41:19] greenrosetta: Now you have *two* admins here :-). [18:42:25] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [18:44:34] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:44:34] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:45:37] DaBPunkt, Nosy: BTW, I believe when LDAP gets flaky, I receive "Lost connection to MySQL server" errors in my scripts. I seem to remember that connections to MySQL are balanced via the HA hosts. Is that correct so that when one of them gets down, the connections with them get down as well? [18:46:05] the connection will be closed, yes [18:46:35] it is way too complicated to implemate a moving of open ports [18:47:09] aliasd on yarrow is OK: TCP OK - 1.734 second response time on port 984 [500 Not found.] [18:47:09] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [18:47:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:47:19] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:47:38] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:47:48] MySQL slave on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [18:47:59] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:47:59] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:07] Probably :-). Let's see if the LDAP failures will continue after Nosy updates the Solaris hosts on Friday. [18:48:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:09] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:48:19] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:48:38] / on thyme is UNKNOWN: NRPE: Unable to read output [18:48:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:48:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:48:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:48:58] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:58] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:08] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:49:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2160229.000000 [18:49:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:49:09] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:49:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:49:10] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [18:49:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 948173.000000 [18:49:19] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [18:49:19] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [18:49:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 790963.000000 [18:49:19] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:49:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 553756.000000 [18:49:20] SSH on mayapple is CRITICAL: Server answer: [18:49:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:49:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1371446.000000 [18:49:39] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [18:49:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [18:49:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:39] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [18:49:39] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:39] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:49:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:49] MySQL on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [18:49:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:49] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:49:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 250055 MB (4% inode=62%): [18:49:50] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:51] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [18:49:51] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:52] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:50:09] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:09] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:09] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:09] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:09] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:50:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:19] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44136 MB (7% inode=99%): [18:50:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:50:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368440.000000 [18:50:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:50:48] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:51:19] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:57:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:05:08] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:06:28] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 85473 MB (8% inode=99%): [19:19:59] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:19] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2167.000000 [19:22:50] MySQL on z-dat-s5-b is OK: Uptime: 15 Threads: 7 Questions: 97 Slow queries: 0 Opens: 39 Flush tables: 1 Open tables: 31 Queries per second avg: 6.466 [19:26:49] MySQL on z-dat-s2-b is OK: Uptime: 11 Threads: 2 Questions: 14 Slow queries: 0 Opens: 596 Flush tables: 1 Open tables: 8 Queries per second avg: 1.272 [19:44:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [19:46:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:19] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:46:19] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:19] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:46:19] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:46:19] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:46:19] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:46:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:20] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:47:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:47:18] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:47:19] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:19] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:19] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:19] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:47:19] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:20] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:47:20] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:21] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3169.000000 [19:47:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:47:22] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:47:22] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:23] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:38] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:47:59] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:47:59] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:09] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:19] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91207 [19:48:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:48:19] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:48:39] / on thyme is UNKNOWN: NRPE: Unable to read output [19:48:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:48:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:48:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:48:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:08] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:49:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:49:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2163802.000000 [19:49:09] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:49:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:49:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 951773.000000 [19:49:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 789719.000000 [19:49:19] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:49:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 557146.000000 [19:49:19] SSH on mayapple is CRITICAL: Server answer: [19:49:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:49:29] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39802.000000 [19:49:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1374982.000000 [19:49:39] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91193.000000 [19:49:39] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [19:49:39] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:49:39] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [19:49:40] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:49:49] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1310609.000000 [19:49:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:49] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:49:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 249690 MB (4% inode=62%): [19:49:50] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:51] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:51] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:52] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:49:59] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91774 [19:49:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:49:59] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:49:59] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:00] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:50:09] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:09] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:09] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:09] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:10] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:50:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:22] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:22] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43908 MB (7% inode=99%): [19:50:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:50:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 371552.000000 [19:50:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:50:48] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:51:19] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1609.000000 [19:51:19] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:53:48] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/7:UP:1 UP: OK [19:53:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/10:UP:1 UP: OK [19:53:49] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/2:UP:1 UP: OK [19:53:59] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/8:UP:1 UP: OK [19:53:59] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/11:UP:1 UP: OK [19:53:59] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/20:UP:1 UP: OK [19:53:59] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/12:UP:1 UP: OK [19:53:59] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/21:UP:1 UP: OK [19:54:00] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/9:UP:1 UP: OK [19:54:08] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/13:UP:1 UP: OK [19:54:09] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/3:UP:1 UP: OK [19:54:09] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/18:UP:1 UP: OK [19:54:09] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/16:UP:1 UP: OK [19:54:09] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/4:UP:1 UP: OK [19:54:09] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/15:UP:1 UP: OK [19:54:10] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/17:UP:1 UP: OK [19:54:10] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/24:UP:1 UP: OK [19:54:11] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/6:UP:1 UP: OK [19:54:11] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/1:UP:1 UP: OK [19:54:12] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/22:UP:1 UP: OK [19:54:12] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/19:UP:1 UP: OK [19:54:13] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/23:UP:1 UP: OK [19:54:13] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/14:UP:1 UP: OK [19:54:38] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is OK: GigabitEthernet0/1/5:UP:1 UP: OK [19:57:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:59:09] aliasd on yarrow is CRITICAL: Connection refused [20:05:18] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:09:09] aliasd on yarrow is OK: TCP OK - 0.004 second response time on port 984 [500 Not found.] [20:11:48] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:12:18] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:17:03] Sumana Harihareswara * [Toolserver-l] Submit OKCon talks by 24 May - travel subsidies available [20:19:59] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:49] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.009 second response time [20:46:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:19] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:29] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:47:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:47:18] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:47:28] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:47:29] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:47:29] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:47:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:47:49] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:47:59] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:47:59] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:09] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:38] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90296 [20:48:38] / on thyme is UNKNOWN: NRPE: Unable to read output [20:48:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:48:48] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:48:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:48:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:08] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:49:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:49:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2166877.000000 [20:49:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 955373.000000 [20:49:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:49:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 789410.000000 [20:49:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 554246.000000 [20:49:19] SSH on mayapple is CRITICAL: Server answer: [20:49:20] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:49:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:49:29] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18684.000000 [20:49:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1375254.000000 [20:49:39] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90332.000000 [20:49:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:39] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:39] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:49:39] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [20:49:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [20:49:40] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:49:49] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1313091.000000 [20:49:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:49] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:49] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:49:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 249340 MB (4% inode=62%): [20:49:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:49:59] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91306 [20:49:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:50:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:50:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:29] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:29] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:29] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:30] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:30] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:31] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43617 MB (7% inode=99%): [20:50:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:50:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 366547.000000 [20:50:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:50:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:48] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:29] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:57:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:01:59] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:18] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:34:49] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:29] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:40:48] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:08] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:18] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:29] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:47:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:47:19] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:47:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:47:28] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:47:28] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:47:28] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:47:48] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:47:58] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:47:59] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:09] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:39] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89480 [21:48:39] / on thyme is UNKNOWN: NRPE: Unable to read output [21:48:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:48:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:48:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:48:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:08] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:49:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:19] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 958972.000000 [21:49:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:49:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 786977.000000 [21:49:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 553359.000000 [21:49:19] SSH on mayapple is CRITICAL: Server answer: [21:49:28] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:49:28] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2166869.000000 [21:49:29] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5878.000000 [21:49:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1375225.000000 [21:49:38] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89335.000000 [21:49:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:39] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:49:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [21:49:39] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [21:49:39] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:40] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:49:49] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1314420.000000 [21:49:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:49] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:49] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:49:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 249150 MB (4% inode=62%): [21:49:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:49:59] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 80441 [21:49:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:50:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:50:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:29] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:29] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:29] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:29] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:30] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:30] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43299 MB (7% inode=99%): [21:50:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:50:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 365839.000000 [21:50:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:50:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:28] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:55:29] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2971.000000 [21:57:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:58:28] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1226.000000 [22:00:18] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:01:58] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:48] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:36:19] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:40:28] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [22:41:31] yarrow is slow again. [22:41:39] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:41:49] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:42:28] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [22:43:19] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:08] aliasd on yarrow is OK: TCP OK - 0.004 second response time on port 984 [500 Not found.] [22:46:08] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:18] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:29] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:19] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:47:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:47:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:29] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:29] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:29] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:48] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:47:58] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:47:59] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:09] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:39] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 76243 [22:48:39] / on thyme is UNKNOWN: NRPE: Unable to read output [22:48:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:48:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:48:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:48:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:48:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:08] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:49:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:49:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:49:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 962573.000000 [22:49:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 785371.000000 [22:49:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 552057.000000 [22:49:19] SSH on mayapple is CRITICAL: Server answer: [22:49:28] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:49:29] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2166217.000000 [22:49:29] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:49:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1375696.000000 [22:49:38] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75964.000000 [22:49:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:39] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:49:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [22:49:39] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [22:49:39] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:40] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:49:49] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1316284.000000 [22:49:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:49] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:49] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:49:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 248976 MB (4% inode=62%): [22:49:59] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:49:59] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58190 [22:49:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:50:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:50:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:29] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:29] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:29] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:30] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:30] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:31] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43050 MB (7% inode=99%): [22:50:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:50:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 365977.000000 [22:50:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:50:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:48] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:28] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:57:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:01:19] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:01:48] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [23:35:48] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:36:19] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:40:58] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:49] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [23:45:55] nacht ts [23:45:58] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:08] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:08] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:19] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:28] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:29] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:47:19] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:47:29] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:29] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:29] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:48] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:47:58] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:47:58] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:08] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:08] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:39] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61968 [23:48:39] / on thyme is UNKNOWN: NRPE: Unable to read output [23:48:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:48:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:48:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:48:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:49:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:49:19] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 966172.000000 [23:49:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:49:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 783563.000000 [23:49:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 551694.000000 [23:49:19] SSH on mayapple is CRITICAL: Server answer: [23:49:28] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:49:28] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2167558.000000 [23:49:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:49:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1375949.000000 [23:49:38] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61980.000000 [23:49:38] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [23:49:39] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:49:39] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [23:49:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:40] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:49:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1318069.000000 [23:49:49] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:49] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:49:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 252033 MB (4% inode=62%): [23:49:58] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:49:59] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44283 [23:49:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:59] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:50:08] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:50:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:29] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:29] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:29] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:29] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:29] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:30] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42724 MB (6% inode=99%): [23:50:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 366532.000000 [23:50:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:50:50] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:50] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:28] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:57:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output