[00:09:51] nope [00:10:00] [[Ghel]] M 10https://wiki.toolserver.org/w/index.php?diff=7999&oldid=7998&rcid=21892 * Dispenser * (+2361) (Reverted edits by [[Special:Contributions/88.64.141.88|88.64.141.88]] ([[User talk:88.64.141.88|talk]]) to last revision by [[User:109.254.49.5|109.254.49.5]]) [00:10:31] k [00:11:24] / on ortelius is WARNING: DISK WARNING - free space: / 4895 MB (16% inode=92%): [00:14:23] / on hemlock is WARNING: DISK WARNING - free space: / 4186 MB (20% inode=88%): [00:15:40] /tmp on hemlock is WARNING: DISK WARNING - free space: / 4186 MB (20% inode=88%): [00:20:22] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:12] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [00:21:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:21:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:22:40] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:23:01] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:23:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36811 MB (6% inode=99%): [00:23:41] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1054636.000000 [00:23:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 353806.000000 [00:23:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:23:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:23:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:23:53] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:23:54] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [00:24:01] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:01] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2237914.000000 [00:24:23] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:24:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:24:33] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:33] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:24:33] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:24:33] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:24:33] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:24:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:34] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:34] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:35] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:35] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:24:54] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:54] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [00:25:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:25:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:25:32] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:32] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:33] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:33] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:33] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:25:33] SSH on mayapple is CRITICAL: Server answer: [00:25:34] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 522326.000000 [00:25:34] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1305115.000000 [00:25:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:53] / on thyme is UNKNOWN: NRPE: Unable to read output [00:25:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1353482.000000 [00:25:53] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:01] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:26:02] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:13] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:33] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:33] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:33] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:54] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:54] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:54] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:54] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:12] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:27:54] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:31:01] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [00:31:01] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:31:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:31:33] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:32:01] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:33:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:33:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:33:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:34:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:34:32] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14190.000000 [00:35:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 286240 MB (5% inode=64%): [00:35:12] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [00:35:41] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:38:12] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:40:33] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:22] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.008 second response time [01:11:22] / on ortelius is WARNING: DISK WARNING - free space: / 4688 MB (15% inode=92%): [01:14:23] / on hemlock is WARNING: DISK WARNING - free space: / 4184 MB (20% inode=88%): [01:15:40] /tmp on hemlock is WARNING: DISK WARNING - free space: / 4184 MB (20% inode=88%): [01:21:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:22:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:23:12] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:23:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36626 MB (5% inode=99%): [01:23:40] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1058236.000000 [01:23:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354142.000000 [01:23:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:23:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:23:54] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:23:54] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:24:02] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [01:24:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:24:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:24:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2241191.000000 [01:24:23] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:24:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:24:40] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:24:41] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:24:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 721387.000000 [01:24:41] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:24:41] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:24:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:24:54] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:24:54] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:24:54] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:24:54] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [01:25:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:25:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:25:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:25:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:25:32] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:32] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:32] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:25:33] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:25:33] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:25:34] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:25:34] SSH on mayapple is CRITICAL: Server answer: [01:25:35] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:25:35] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:25:36] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:25:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1306417.000000 [01:25:54] / on thyme is UNKNOWN: NRPE: Unable to read output [01:25:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1355141.000000 [01:25:54] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:13] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:32] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:33] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:33] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:53] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:53] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:54] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:27:01] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:27:12] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:27:41] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1992.000000 [01:27:54] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1993 [01:27:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:54] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:31:00] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [01:31:01] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:32:01] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [01:32:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:32:32] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:33:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:33:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:33:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:34:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:34:33] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16436.000000 [01:35:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289288 MB (5% inode=64%): [01:35:40] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:35:54] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [01:36:13] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:38:40] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:40:22] Load avg. on nightshade is WARNING: WARNING - load average: 24.21, 18.83, 9.73 [01:42:22] Load avg. on nightshade is OK: OK - load average: 6.79, 14.48, 9.27 [01:51:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:58:12] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:11:22] / on ortelius is WARNING: DISK WARNING - free space: / 4419 MB (14% inode=92%): [02:14:23] / on hemlock is WARNING: DISK WARNING - free space: / 4181 MB (20% inode=88%): [02:15:41] /tmp on hemlock is WARNING: DISK WARNING - free space: / 4181 MB (20% inode=88%): [02:21:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:22:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:23:12] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:23:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34199 MB (5% inode=99%): [02:23:40] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1061836.000000 [02:23:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354652.000000 [02:24:02] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [02:24:13] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:24:13] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:24:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2244421.000000 [02:24:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 719802.000000 [02:24:41] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:24:41] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:24:41] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:24:41] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:24:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:24:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:24:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:24:53] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:24:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:24:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:01] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [02:25:02] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:25:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:25:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:25:22] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:25:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 522129.000000 [02:25:42] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:42] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:42] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:43] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:43] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:44] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:44] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:45] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:54] / on thyme is UNKNOWN: NRPE: Unable to read output [02:25:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1357841.000000 [02:25:54] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:54] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:01] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:26:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:26:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:26:33] SSH on mayapple is CRITICAL: Server answer: [02:26:33] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:26:33] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:34] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:34] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:35] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:35] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:41] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1305604.000000 [02:26:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:41] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:53] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:01] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [02:27:02] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:02] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:13] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:41] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2916.000000 [02:27:54] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2915 [02:27:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:54] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:28:12] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:28:26] @replag [02:28:28] Betacommand: s1-rr-a: 21h 45m 59s [+1.00 s/s]; s1-rr-a-wd: 1w 4d 3h 58m 26s [+0.04 s/s]; s1-user: 48m 50s [-]; s1-user-c: 1w 5d 7h 2m 9s [+1.00 s/s]; s1-user-wd: 2w 1d 17h 12m 34s [-0.36 s/s]; s2-rr: 21h 46m 54s [+1.00 s/s]; s2-user: 21h 46m 54s [+1.00 s/s]; s2-user-c: error [02:28:29] Betacommand: s2-user-wd: 2w 1d 2h 39m 49s [-0.17 s/s]; s3-user: 29s [+0.00 s/s]; s3-user-wd: 1w 1d 14h 37m 24s [-0.80 s/s]; s4-user-wd: 3w 4d 23h 30m 49s [+0.84 s/s]; s5-rr-a: 5h 7m 41s [+0.60 s/s]; s5-rr-a-wd: 4d 2h 31m 27s [+0.07 s/s]; s5-user: 21h 52m 57s [+1.00 s/s]; s6-user-wd: 1w 1d 7h 56m 29s [-0.68 s/s] [02:28:30] Betacommand: s7-user-wd: 6d 1h 1m 57s [-0.02 s/s] [02:31:01] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [02:31:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:32:01] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [02:32:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:32:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:33:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:33:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:33:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:34:33] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18639.000000 [02:35:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293515 MB (5% inode=64%): [02:35:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:36:31] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [02:36:41] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:12] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:51:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:58:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3613.000000 [02:58:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3619 [03:15:00] [[Special:Log/newusers]] create 10 * Milliondollar * (New user account) [03:21:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:22:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:23:11] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:23:41] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36462 MB (5% inode=99%): [03:23:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1065436.000000 [03:23:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354924.000000 [03:24:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:24:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2247458.000000 [03:24:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 718614.000000 [03:24:41] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:24:41] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:24:42] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:24:42] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:24:42] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:24:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:24:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:24:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:24:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:24:55] [[User talk:Milliondollar]] !N 10https://wiki.toolserver.org/w/index.php?oldid=8000&rcid=21894 * Milliondollar * (+133) (Created page with "[http://dollarover.skyrock.com/3160822404-Million-Dollar-Makeovers-For-A-Valuable-Residential-property.html million dollar makeovers]") [03:25:00] bleh [03:25:01] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:25:01] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:25:01] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [03:25:11] [[Special:Log/block]] block 10 * Legoktm * (blocked [[02User:Milliondollar10]] with an expiry time of infinite (account creation disabled): spam) [03:25:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:25:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:25:13] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:25:22] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:25:27] [[Special:Log/delete]] delete 10 * Legoktm * (deleted "[[02User talk:Milliondollar10]]": spam) [03:25:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 521423.000000 [03:25:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:42] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:42] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:42] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:43] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:43] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:44] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:44] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:45] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:45] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:53] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:01] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [03:26:01] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:26:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:26:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:41] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:41] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:41] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:42] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1305233.000000 [03:26:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:43] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:53] / on thyme is UNKNOWN: NRPE: Unable to read output [03:26:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1360286.000000 [03:26:54] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:54] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:01] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [03:27:02] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:02] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:12] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:27:32] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:27:33] SSH on mayapple is CRITICAL: Server answer: [03:28:12] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:28:53] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:54] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:31:01] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [03:32:01] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [03:32:01] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:32:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:32:32] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:33:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:33:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:33:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:35:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:35:32] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20527.000000 [03:36:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293341 MB (5% inode=64%): [03:36:42] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:12] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [03:37:12] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:40:32] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:42:42] Load avg. on nightshade is WARNING: WARNING - load average: 15.53, 17.44, 10.69 [03:45:41] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:45:53] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:32] Load avg. on nightshade is WARNING: WARNING - load average: 28.63, 21.05, 13.45 [03:51:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:55:41] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3595.000000 [03:55:53] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3589 [04:10:12] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:11:21] / on ortelius is WARNING: DISK WARNING - free space: / 5732 MB (19% inode=92%): [04:17:32] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.018 second response time [04:21:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:22:52] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:23:11] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:23:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36392 MB (5% inode=99%): [04:23:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1069036.000000 [04:23:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354902.000000 [04:24:23] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:24:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2250560.000000 [04:24:41] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:24:41] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:24:41] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:24:41] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:24:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:24:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:25:01] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:25:02] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [04:25:02] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:25:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:25:23] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:25:23] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:25:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 520712.000000 [04:25:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 717653.000000 [04:25:42] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:42] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:43] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:43] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:44] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:44] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:45] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:45] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:46] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:53] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:53] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:25:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:53] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:54] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:54] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:26:01] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [04:26:12] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:26:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:26:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:26:31] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [04:26:41] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1302879.000000 [04:26:42] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:26:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:26:43] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:26:43] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:26:43] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:26:43] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:26:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:26:43] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:02] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [04:27:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:12] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:23] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:23] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:27:42] SSH on mayapple is CRITICAL: Server answer: [04:27:52] / on thyme is UNKNOWN: NRPE: Unable to read output [04:27:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1361183.000000 [04:27:52] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:28:12] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:28:30] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:28:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:28:53] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:31:01] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [04:32:01] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [04:32:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:32:13] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:33:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:34:52] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:34:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:34:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:35:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:36:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293217 MB (5% inode=64%): [04:36:31] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22388.000000 [04:36:41] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:37:13] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:37:52] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [04:43:41] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 84506 MB (8% inode=99%): [04:51:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:55:41] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3112.000000 [04:55:53] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3115 [05:11:23] / on ortelius is WARNING: DISK WARNING - free space: / 5566 MB (18% inode=92%): [05:15:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [05:15:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3605 [05:16:53] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3596.000000 [05:16:53] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3596 [05:17:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3616.000000 [05:17:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3619 [05:21:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:23:02] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:23:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:23:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354449.000000 [05:23:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1072643.000000 [05:24:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2253579.000000 [05:24:31] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:24:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36299 MB (5% inode=99%): [05:25:03] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:25:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:25:03] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:25:03] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:25:03] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:25:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:25:12] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:25:12] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [05:25:22] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:25:32] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:32] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:26:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:02] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:02] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:02] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:03] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:03] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:04] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:04] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:05] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:06] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:06] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:07] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:21] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:26:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:26:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 519838.000000 [05:26:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 716364.000000 [05:26:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1300300.000000 [05:27:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:27:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:27:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:27:03] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:27:03] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:27:04] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:27:04] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:27:05] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:27:05] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:27:06] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:27:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:31] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:31] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:31] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:31] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:53] / on thyme is UNKNOWN: NRPE: Unable to read output [05:28:02] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:28:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1363301.000000 [05:28:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:28:40] SSH on mayapple is CRITICAL: Server answer: [05:28:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:29:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:29:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:29:11] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:31:02] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [05:32:02] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [05:32:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:33:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:33:41] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:34:40] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:34:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:34:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:34:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:35:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:36:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293043 MB (5% inode=64%): [05:36:41] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23974.000000 [05:37:22] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:38:32] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [05:42:53] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3599.000000 [05:42:53] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3595 [05:51:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:58:31] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.009 second response time [06:02:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [06:02:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3616 [06:12:21] / on ortelius is WARNING: DISK WARNING - free space: / 5211 MB (17% inode=92%): [06:21:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:23:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:23:42] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:23:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 353973.000000 [06:23:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1076244.000000 [06:24:42] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:24:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36138 MB (5% inode=99%): [06:25:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:25:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:25:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2256678.000000 [06:25:22] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:25:22] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [06:25:41] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:25:41] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:25:41] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:26:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:26:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:03] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:03] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:04] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:04] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:13] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:13] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:26:22] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [06:26:23] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:26:41] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:26:41] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:26:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518669.000000 [06:26:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715925.000000 [06:27:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:27:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:27:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:27:02] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:27:12] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:27:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:27:12] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:27:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:27:12] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:27:41] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:27:42] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:27:42] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:27:42] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:27:42] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:27:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1296633.000000 [06:27:52] / on thyme is UNKNOWN: NRPE: Unable to read output [06:28:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:03] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:28:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:03] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1366444.000000 [06:28:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:28:41] SSH on mayapple is CRITICAL: Server answer: [06:28:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:29:12] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:29:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:29:22] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:31:01] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:32:02] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:33:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:33:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:33:41] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:34:53] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:35:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:35:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:35:52] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:35:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:36:33] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:36:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25679.000000 [06:37:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292848 MB (5% inode=64%): [06:38:53] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:39:11] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [06:40:32] Load avg. on nightshade is WARNING: WARNING - load average: 21.36, 15.29, 8.36 [06:41:52] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 30.44, 19.52, 10.41 [06:44:41] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:44:41] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:32] aliasd on nightshade is OK: TCP OK - 8.653 second response time on port 984 [500 Not found.] [06:45:32] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [06:46:32] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [06:47:32] Load avg. on nightshade is WARNING: WARNING - load average: 13.11, 24.35, 16.22 [06:51:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:02:32] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:03:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4437.000000 [07:03:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4437 [07:12:22] / on ortelius is WARNING: DISK WARNING - free space: / 4775 MB (15% inode=92%): [07:22:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:23:13] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:23:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 353788.000000 [07:23:53] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:23:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1079847.000000 [07:24:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35942 MB (5% inode=99%): [07:24:52] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:25:13] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:25:22] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:25:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2259901.000000 [07:25:22] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:25:22] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [07:25:53] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:25:53] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:26:02] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:26:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:26:13] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:26:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:26:13] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:26:13] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:14] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:15] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:15] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:16] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:16] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:17] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:17] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:23] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:26:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:26:23] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [07:26:33] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:26:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 517842.000000 [07:26:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715128.000000 [07:26:53] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:26:53] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:27:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:27:12] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:12] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:13] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:21] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:22] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:22] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:22] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1293846.000000 [07:27:53] / on thyme is UNKNOWN: NRPE: Unable to read output [07:27:53] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:27:53] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:27:53] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:27:54] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:27:54] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:28:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1369703.000000 [07:28:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:12] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:12] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:13] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:41] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:28:41] SSH on mayapple is CRITICAL: Server answer: [07:29:01] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:29:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:23] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:29:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:31:02] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [07:33:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:33:02] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:33:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:33:42] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:35:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:36:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:36:41] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28399.000000 [07:36:52] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:36:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:37:01] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292524 MB (5% inode=64%): [07:38:41] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:39:52] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [07:40:31] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [07:41:01] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:41:32] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 34.79, 23.32, 13.08 [07:42:31] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [07:43:31] Load avg. on nightshade is WARNING: WARNING - load average: 15.21, 21.57, 13.82 [07:46:32] Load avg. on nightshade is OK: OK - load average: 4.95, 13.81, 12.16 [08:00:32] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:04:51] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4289 [08:04:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4290.000000 [08:12:23] / on ortelius is WARNING: DISK WARNING - free space: / 4443 MB (14% inode=92%): [08:16:01] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [08:17:02] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [08:21:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:22:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:23:22] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:24:02] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:24:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 352012.000000 [08:24:51] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1083507.000000 [08:25:11] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:25:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:25:32] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:25:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2263161.000000 [08:25:32] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:25:32] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [08:25:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35706 MB (5% inode=99%): [08:26:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:26:12] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:26:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:26:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:26:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:23] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:26:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:25] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:26] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:27] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:27] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:26:28] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:27:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:27:12] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:27:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:23] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:24] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:24] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:25] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:31] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:27:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518252.000000 [08:27:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713447.000000 [08:27:52] / on thyme is UNKNOWN: NRPE: Unable to read output [08:28:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:11] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:12] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1373061.000000 [08:28:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:23] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:23] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:24] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:42] SSH on mayapple is CRITICAL: Server answer: [08:28:52] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:28:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [08:29:02] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 99877 [08:29:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:30] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:29:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:32:46] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [08:33:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:33:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:34:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:34:02] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [08:35:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:36:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:36:43] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31256.000000 [08:36:52] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:36:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:38:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292268 MB (5% inode=64%): [08:40:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100290.000000 [08:47:22] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2198.000000 [08:53:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74860 MB (12% inode=99%): [08:58:01] MySQL on z-dat-s2-b is OK: Uptime: 2890 Threads: 7 Questions: 940 Slow queries: 0 Opens: 166 Flush tables: 1 Open tables: 158 Queries per second avg: 0.325 [09:00:31] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:04:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3807 [09:04:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3808.000000 [09:13:23] / on ortelius is WARNING: DISK WARNING - free space: / 4123 MB (13% inode=92%): [09:17:01] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [09:18:23] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1527.000000 [09:22:51] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3582 [09:22:51] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3583.000000 [09:23:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:24:12] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:24:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 350514.000000 [09:24:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1087109.000000 [09:25:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:26:12] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:22] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:31] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:31] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:32] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:32] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:32] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:33] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2266503.000000 [09:26:33] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:26:34] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [09:26:34] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [09:27:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:27:22] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:27:22] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:24] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:24] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:31] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:31] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:32] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:27:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:27:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518091.000000 [09:27:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 711516.000000 [09:28:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1376184.000000 [09:28:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:13] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:13] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:13] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:13] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:14] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:21] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:28:43] SSH on mayapple is CRITICAL: Server answer: [09:28:52] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:28:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1296293.000000 [09:28:53] / on thyme is UNKNOWN: NRPE: Unable to read output [09:29:01] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98715 [09:29:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:23] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:23] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:23] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:29:31] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:29:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:30:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:32:02] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [09:33:11] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:33:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:34:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:34:02] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92103 [09:35:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:35:44] [[Special:Log/newusers]] create 10 * Gingping121 * (New user account) [09:36:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:37:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34105.000000 [09:37:52] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:37:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:38:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291905 MB (5% inode=64%): [09:39:02] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:43] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:41:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98016.000000 [09:42:52] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:43:01] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:44:32] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [09:44:42] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 36.98, 25.58, 13.98 [09:51:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:53:43] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67343 MB (11% inode=99%): [09:59:52] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:59:52] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:03:43] aliasd on nightshade is OK: TCP OK - 8.824 second response time on port 984 [500 Not found.] [10:06:32] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.016 second response time [10:09:52] MySQL slave on rosemary is OK: Uptime: 296087 Threads: 75 Questions: 274788961 Slow queries: 84178 Opens: 21337 Flush tables: 1 Open tables: 3133 Queries per second avg: 928.68 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1772 [10:09:52] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1771.000000 [10:14:21] / on ortelius is WARNING: DISK WARNING - free space: / 3561 MB (11% inode=92%): [10:17:02] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:23:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:23:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:24:12] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:52] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348884.000000 [10:25:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1090714.000000 [10:25:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:25:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:26:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:26:11] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:26:11] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:26:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:26:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:33] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:33] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:33] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:33] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:33] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2269738.000000 [10:26:34] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:26:34] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [10:26:35] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [10:27:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:27:21] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:21] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:22] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:22] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:32] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:32] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:33] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:33] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:34] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:34] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:27:36] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:27:51] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518413.000000 [10:27:51] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 708731.000000 [10:28:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1378681.000000 [10:28:12] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:28:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:28:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:28:44] SSH on mayapple is CRITICAL: Server answer: [10:28:52] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:29:01] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88430 [10:29:11] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:29:11] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:29:12] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:29:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:29:12] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:29:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:29:32] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:29:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:29:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1296962.000000 [10:29:52] / on thyme is UNKNOWN: NRPE: Unable to read output [10:30:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:22] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:22] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:33:01] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:33:11] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:34:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:34:02] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82756 [10:34:21] aliasd on nightshade is CRITICAL: Connection refused [10:34:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:35:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:36:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:37:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36151.000000 [10:37:52] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:37:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:38:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291588 MB (5% inode=64%): [10:41:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85903.000000 [10:45:52] /tmp on hemlock is WARNING: DISK WARNING - free space: / 4193 MB (20% inode=88%): [10:46:02] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 32.46, 33.74, 37.73 [10:46:22] / on hemlock is WARNING: DISK WARNING - free space: / 4193 MB (20% inode=88%): [10:51:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:52:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67091 MB (10% inode=99%): [10:54:22] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 3294 MB (10% inode=92%): [10:55:21] / on ortelius is WARNING: DISK WARNING - free space: / 3300 MB (11% inode=92%): [10:58:22] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 3290 MB (10% inode=92%): [11:17:01] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:23:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:23:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:24:23] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:24:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347864.000000 [11:25:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1094314.000000 [11:25:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:26:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:26:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:26:23] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:26:23] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:26:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:26:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:26:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:26:32] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:27:21] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:21] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:22] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:32] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:32] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:33] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2272974.000000 [11:27:40] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:40] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:27:40] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [11:27:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [11:27:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518986.000000 [11:27:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 705039.000000 [11:28:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1381787.000000 [11:28:12] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:23] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:28:31] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:28:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:28:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:29:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:29:02] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 71238 [11:29:22] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:29:23] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:29:23] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:29:23] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:29:23] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:29:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:29:32] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:29:43] SSH on mayapple is CRITICAL: Server answer: [11:29:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:29:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1297267.000000 [11:29:52] / on thyme is UNKNOWN: NRPE: Unable to read output [11:30:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:22] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:22] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:33:02] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:33:52] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:02] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:34:02] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 70171 [11:34:11] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:34:22] aliasd on nightshade is CRITICAL: Connection refused [11:34:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:35:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:37:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38561.000000 [11:37:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:37:51] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:37:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:38:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291047 MB (5% inode=64%): [11:38:23] / on ortelius is OK: DISK OK - free space: / 7874 MB (26% inode=92%): [11:40:43] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.454 second response time [11:41:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68364.000000 [11:44:22] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [11:45:52] /tmp on hemlock is WARNING: DISK WARNING - free space: / 4190 MB (20% inode=88%): [11:46:22] / on hemlock is WARNING: DISK WARNING - free space: / 4190 MB (20% inode=88%): [11:46:43] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 21.66, 22.07, 27.16 [11:52:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66775 MB (10% inode=99%): [11:55:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:17:01] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:21:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:23:41] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:23] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:24:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347962.000000 [12:25:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1097923.000000 [12:25:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:26:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:26:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:26:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:26:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:27:23] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:23] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:23] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:31] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:42] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:43] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:43] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:43] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:44] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:44] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:45] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518994.000000 [12:27:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 704472.000000 [12:28:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1385202.000000 [12:28:22] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:28:22] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:28:31] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:28:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:28:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:28:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:28:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:28:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2276239.000000 [12:28:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:28:33] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [12:28:33] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [12:28:34] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:28:34] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:28:35] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:28:41] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:29:02] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:29:23] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:29:23] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:29:23] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:29:23] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:29:23] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:29:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:29:32] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:29:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:29:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1299424.000000 [12:29:53] / on thyme is UNKNOWN: NRPE: Unable to read output [12:30:01] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63535 [12:30:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:23] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:23] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:23] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:31] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:30:42] SSH on mayapple is CRITICAL: Server answer: [12:30:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:33:12] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:34:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:34:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 69067 [12:34:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:34:23] aliasd on nightshade is CRITICAL: Connection refused [12:35:01] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:35:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:37:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:37:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:38:41] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40886.000000 [12:38:41] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:39:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 290710 MB (5% inode=64%): [12:41:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63307.000000 [12:43:41] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:43:52] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:44:12] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:44:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:45:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:45:33] can some kill/restart nscd on nightshade? https://jira.toolserver.org/browse/TS-1648 http://unix.stackexchange.com/questions/69269/nscd-uses-100-cpu http://bugs.centos.org/view.php?id=3373 (first google results for lazies) [12:45:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:45:53] /tmp on hemlock is WARNING: DISK WARNING - free space: / 4187 MB (20% inode=88%): [12:46:23] / on hemlock is WARNING: DISK WARNING - free space: / 4187 MB (20% inode=88%): [12:47:42] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 66.81, 70.14, 63.04 [12:48:53] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:11] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:52:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66506 MB (10% inode=99%): [13:07:41] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.149 second response time [13:08:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:14:52] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:17:01] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [13:21:41] Load avg. on nightshade is WARNING: WARNING - load average: 2.58, 4.19, 19.68 [13:23:43] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:24:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348112.000000 [13:25:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1101524.000000 [13:25:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:26:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:26:21] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:26:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:26:41] Load avg. on nightshade is OK: OK - load average: 1.87, 2.72, 14.75 [13:26:42] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:22] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:27:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:32] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 514123.000000 [13:28:01] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 701611.000000 [13:28:22] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:28:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:28:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:23] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:33] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:33] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:33] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:33] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2279528.000000 [13:28:33] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:28:34] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [13:28:34] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [13:28:35] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:28:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:42] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:42] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:42] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:42] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:43] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:44] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:44] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:28:45] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:29:22] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:29:22] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:29:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1388794.000000 [13:29:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:29:32] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:30:01] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:30:01] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57453 [13:30:22] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:30:22] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:30:22] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:30:23] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:30:23] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:30:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:23] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:24] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:24] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:32] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:30:42] SSH on mayapple is CRITICAL: Server answer: [13:30:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:30:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1300356.000000 [13:30:53] / on thyme is UNKNOWN: NRPE: Unable to read output [13:33:12] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [13:34:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:34:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 66419 [13:34:13] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:35:01] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:35:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:35:23] aliasd on nightshade is CRITICAL: Connection refused [13:37:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:38:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:38:41] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43218.000000 [13:38:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:39:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 290262 MB (5% inode=64%): [13:41:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56069.000000 [13:44:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:45:53] /tmp on hemlock is WARNING: DISK WARNING - free space: / 4184 MB (20% inode=88%): [13:46:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:46:31] / on hemlock is WARNING: DISK WARNING - free space: / 4184 MB (20% inode=88%): [13:49:32] / on hemlock is OK: DISK OK - free space: / 8154 MB (40% inode=88%): [13:49:53] /tmp on hemlock is OK: DISK OK - free space: / 8154 MB (40% inode=88%): [13:51:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:52:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66224 MB (10% inode=99%): [14:09:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:14:52] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [14:17:02] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [14:24:02] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:24:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 349429.000000 [14:25:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1105133.000000 [14:25:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:26:33] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:26:33] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:02] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:02] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3567 [14:27:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:27:32] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:27:32] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:02] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 700158.000000 [14:28:03] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46173 [14:28:33] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:33] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:33] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:33] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:28:33] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:34] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:34] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:35] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:35] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:28:36] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:36] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:37] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [14:28:37] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2282830.000000 [14:28:38] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [14:28:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512621.000000 [14:29:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:03] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:03] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:03] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:04] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:04] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:05] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:05] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:29:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:29:33] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:29:33] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [14:30:12] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:30:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1392399.000000 [14:30:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:33] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:33] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:30:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:34] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:30:35] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:30:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:30:36] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:30:36] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:31:41] SSH on mayapple is CRITICAL: Server answer: [14:31:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:31:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1301090.000000 [14:31:52] / on thyme is UNKNOWN: NRPE: Unable to read output [14:33:02] MySQL slave on z-dat-s5-b is OK: Uptime: 22935 Threads: 6 Questions: 31008175 Slow queries: 80 Opens: 5170 Flush tables: 1 Open tables: 255 Queries per second avg: 1352.2 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [14:34:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:34:11] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [14:35:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:35:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:35:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64032 [14:35:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:36:21] aliasd on nightshade is CRITICAL: Connection refused [14:38:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:38:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:38:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45434.000000 [14:38:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:39:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289938 MB (5% inode=64%): [14:42:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45567.000000 [14:44:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:46:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:51:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:53:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65905 MB (10% inode=99%): [15:10:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:14:52] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:18:03] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [15:21:01] MySQL on z-dat-s1-b is OK: Uptime: 23637 Threads: 6 Questions: 5547 Slow queries: 1 Opens: 76 Flush tables: 1 Open tables: 69 Queries per second avg: 0.234 [15:24:02] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:25:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1108735.000000 [15:25:42] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:25:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 350282.000000 [15:26:42] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:27:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:27:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:27:42] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:27:42] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:27:52] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:02] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 698921.000000 [15:28:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:42] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:42] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:28:42] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [15:28:42] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [15:28:43] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:43] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:28:53] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:28:53] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:28:53] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512569.000000 [15:29:02] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:02] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:02] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:03] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:11] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:31] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2285969.000000 [15:29:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:32] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:43] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:43] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:29:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:29:43] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:29:43] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:30:02] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:30:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:30:02] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:30:02] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:30:03] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:30:22] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:30:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1395923.000000 [15:30:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:42] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:43] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:30:43] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:30:43] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:30:43] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:30:43] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:30:44] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:31:31] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:32] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:42] SSH on mayapple is CRITICAL: Server answer: [15:31:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:31:52] / on thyme is UNKNOWN: NRPE: Unable to read output [15:31:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1301162.000000 [15:34:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:34:11] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 122944 [15:35:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:35:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56655 [15:35:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:35:22] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:36:22] aliasd on nightshade is CRITICAL: Connection refused [15:38:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:38:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:38:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47700.000000 [15:38:51] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:39:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289626 MB (5% inode=64%): [15:39:22] / on ortelius is WARNING: DISK WARNING - free space: / 6248 MB (20% inode=92%): [15:42:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47853.000000 [15:44:03] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:44:51] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [15:46:02] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1889 [15:46:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:46:21] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1889.000000 [15:51:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:54:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65591 MB (10% inode=99%): [16:10:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:15:52] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:23:23] anyone know what's up with the GE job handling? looks like lots of queues are in an "alarm" state, and I have a short-running job that's been queued since yesterday morning [16:24:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:24:53] Merlissimo: ^ [16:25:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1112338.000000 [16:25:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 350146.000000 [16:25:52] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:26:51] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:27:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:27:43] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:27:43] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:27:51] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:28:11] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:42] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:28:42] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [16:28:42] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [16:28:42] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:42] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:51] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:51] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:29:02] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:29:12] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 698205.000000 [16:29:12] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:12] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:12] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:12] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2289059.000000 [16:29:39] Didn't Nosy reboot the webservers or something? Hmmm. [16:29:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:29:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:43] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:43] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:44] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:44] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:45] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:29:51] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:29:51] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:29:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512920.000000 [16:29:52] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:30:11] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:30:11] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:30:11] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:30:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:30:11] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:30:31] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:30:31] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1399501.000000 [16:30:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:43] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:43] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:30:51] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:30:52] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:30:52] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:30:52] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:30:52] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:31:33] ortelius: up 1 day 21:49, wolfsbane: up 4:42. Don't know if she wants to keep the queues disabled till the Solaris update she has planned. [16:31:42] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:42] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:42] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:43] SSH on mayapple is CRITICAL: Server answer: [16:31:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:32:02] / on thyme is UNKNOWN: NRPE: Unable to read output [16:32:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1302164.000000 [16:35:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:35:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52797 [16:35:11] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114582 [16:35:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:35:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:36:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:36:43] aliasd on nightshade is CRITICAL: Connection refused [16:38:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:38:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:38:21] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:43] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50106.000000 [16:38:51] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:38:53] "qstat -f -explain E": (inter alia) "queue medium-lx marked QERROR as result of job 1975051's failure at host yarrow.toolserver.org"; "qacct -j 1975051": "failed 11 : before job", "qsub_time Thu Jan 1 00:00:00 1970". This doesn't look right, but probably just a glitch with the LDAP QoS; I try to clear the error state from medium-lx@yarrow and longrun-lx@yarrow. [16:39:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:40:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289176 MB (5% inode=64%): [16:40:22] / on ortelius is WARNING: DISK WARNING - free space: / 5946 MB (19% inode=92%): [16:42:12] Done. [16:42:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50279.000000 [16:42:22] Why does nscd on nightshade take 99% CPU?! [16:43:43] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 82227 MB (8% inode=99%): [16:45:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:46:02] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2767 [16:46:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:46:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2771.000000 [16:54:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65342 MB (10% inode=99%): [16:58:03] Tim Landscheidt * Re: [Toolserver-l] Maintenance: Rebooting ortelius web server [17:08:37] how does one kill all of their mysql processes [17:10:31] Betacommand: Doesn't exiting your program cancel them? [17:10:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:10:44] scfc_de: Not always [17:12:21] Some googling: Connect to the server where the query is run, "SHOW PROCESSLIST;", "KILL $ID;". [17:12:26] Does it work for you? [17:13:52] scfc_de: thats what I just did [17:14:12] * Betacommand had to wait for one of them to die on its own [17:14:26] I hit the max_user_connections [17:15:19] Ah, okay. I think you can only kill your own queries (well, I hope so), and otherwise you have to wait for a root. [17:16:51] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:36] Merlissimo: {medium,longrun}-lx@yarrow are in state "E" again ("qacct -j 1975236": "failed 1 : assumedly before job"). Do you know why? [17:21:28] scfc_de: I took care of it [17:21:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:21:51] scfc_de: I was hoping for a way that I could do it without first connecting to mysql [17:22:41] Betacommand: I don't know of any. [17:24:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:25:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 349464.000000 [17:25:52] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:26:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1115998.000000 [17:26:51] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:27:43] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:27:43] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:27:51] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:28:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:28:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:28:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [17:28:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [17:28:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:51] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:52] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:29:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:29:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 697554.000000 [17:29:12] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:13] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:13] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2291895.000000 [17:29:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:29:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:43] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:43] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512912.000000 [17:29:52] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:29:52] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:30:11] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:30:11] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:30:11] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:30:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:30:31] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1402743.000000 [17:30:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:30:43] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:44] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:44] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:30:44] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:30:44] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:30:44] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:30:44] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:30:50] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:30:51] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:31:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:31:36] anyone have a second to help with a sql query? [17:31:43] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:43] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:43] SSH on mayapple is CRITICAL: Server answer: [17:31:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:31:43] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:51] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:31:51] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:31:52] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:32:02] / on thyme is UNKNOWN: NRPE: Unable to read output [17:32:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1302538.000000 [17:32:12] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:32:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:33:14] Betacommand: What do you need? [17:35:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:35:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37514 [17:35:12] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110452 [17:35:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:35:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:36:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:37:07] scfc_de: pages in ns 6 that have been deleted and still do not exist, yet are linked to via imagelinks [17:37:42] aliasd on nightshade is CRITICAL: Connection refused [17:38:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:38:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:38:19] That will take me more than a second :-). [17:38:43] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52327.000000 [17:39:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:39:03] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:39:46] Betacommand: Isn't this https://en.wikipedia.org/wiki/Wikipedia:Database_reports/Articles_containing_deleted_files ? [17:40:22] / on ortelius is WARNING: DISK WARNING - free space: / 5535 MB (18% inode=92%): [17:41:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289004 MB (5% inode=64%): [17:42:17] Betacommand: http://pastebin.com/8qQwx1fA [17:43:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52447.000000 [17:45:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:46:02] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2793 [17:46:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2797.000000 [17:46:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:47:42] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2163 [17:48:32] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2216.000000 [17:54:17] scfc_de: thanks I was looking for that query just couldnt find it [17:54:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65085 MB (10% inode=99%): [18:02:41] MySQL slave on daphne is OK: Uptime: 1808165 Threads: 52 Questions: 219232726 Slow queries: 397200 Opens: 76727 Flush tables: 1 Open tables: 1895 Queries per second avg: 121.245 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1793 [18:03:31] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1672.000000 [18:16:51] Hi Betacommand [18:16:58] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:58] Did you get my mail? [18:17:18] yeah [18:17:30] And I mean the one with the update query [18:17:31] will do the script this weekend [18:17:34] Thank you [18:18:37] BTW Is there a quick way of checking if a youtube link has been 'nuked' on the YouTube side? [18:20:41] Betacommand: One thing I forgot is that the query (might) return multiple lines per il_to if the file was deleted multiple times. In my script, I thus save il_to to last_il_to after every row, and only print lines if last_il_to != il_to. [18:21:59] Qcoder00: Do you have an example URL? [18:22:34] Not to hand [18:23:13] But in reviewing some YouTube/ Google video I was reinstating carefully, I found some of them had been removed due to copyright claims at YouTube [18:24:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:24:26] You could try if http://gdata.youtube.com/feeds/api/videos/$VIDEO_ID has meaningful info. [18:25:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 349161.000000 [18:26:03] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:26:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1119604.000000 [18:27:03] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:27:56] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:27:56] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:03] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:28:22] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:28:42] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [18:28:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [18:28:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [18:28:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:52] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:52] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:52] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:03] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:29:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:23] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:24] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2294870.000000 [18:29:42] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [18:29:52] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:52] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 510968.000000 [18:29:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:30:03] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:30:03] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:30:05] http://www.youtube.com/watch?v=2-MV_RAwkp0&list=PLF4E6EA20D9A80745 is an example of BLOCKED URL [18:30:11] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 697230.000000 [18:30:11] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:30:11] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:30:22] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:30:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:30:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:30:31] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1405405.000000 [18:30:43] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:30:52] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:30:52] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:52] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:52] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:30:52] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:30:52] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:30:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:31:03] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:31:03] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:31:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:31:42] SSH on mayapple is CRITICAL: Server answer: [18:31:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:31:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:51] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:51] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:02] / on thyme is UNKNOWN: NRPE: Unable to read output [18:32:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1303278.000000 [18:32:03] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:32:03] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:32:03] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:33:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:34:06] If you download http://gdata.youtube.com/feeds/api/videos/2-MV_RAwkp0, there's all the information you need. [18:34:42] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.013 second response time [18:35:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:35:42] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:36:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:36:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:36:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29008 [18:36:11] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103511 [18:36:23] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:36:52] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:37:43] aliasd on nightshade is CRITICAL: Connection refused [18:38:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54271.000000 [18:39:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:39:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:39:23] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:40:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:40:22] / on ortelius is WARNING: DISK WARNING - free space: / 5128 MB (17% inode=92%): [18:42:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 288780 MB (5% inode=64%): [18:42:43] Load avg. on nightshade is WARNING: WARNING - load average: 21.40, 15.34, 8.82 [18:43:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54372.000000 [18:45:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:45:42] Load avg. on nightshade is OK: OK - load average: 5.04, 12.57, 9.09 [18:46:03] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2084 [18:46:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:46:32] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2073.000000 [18:48:56] scfc_de: Thanks ;) [18:51:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:54:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64799 MB (10% inode=99%): [18:58:51] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:01:50] Did someone reset the longrun-lx@yarrow queue? It was displayed as without error some time ago, but now both queues at yarrow are in error state again. [19:18:32] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [19:24:22] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:26:13] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:26:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 349198.000000 [19:27:13] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:27:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1123266.000000 [19:28:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:03] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:13] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:29:04] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:04] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:04] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:04] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:29:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:22] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:22] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:29:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2298085.000000 [19:29:42] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [19:29:42] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:29:42] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [19:30:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 696540.000000 [19:30:13] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:30:13] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:30:14] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:30:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:30:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:30:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:30:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:30:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:30:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:30:23] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:30:24] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:30:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1408954.000000 [19:30:42] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:30:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:30:51] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:51] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:51] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508536.000000 [19:31:02] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:31:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:31:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:31:03] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:31:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:31:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:31:12] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:31:50] SSH on mayapple is CRITICAL: Server answer: [19:31:51] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:32:03] / on thyme is UNKNOWN: NRPE: Unable to read output [19:32:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:03] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1305624.000000 [19:32:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:32:13] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:32:13] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:32:22] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:32:51] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:35:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:35:32] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:35:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:36:03] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:36:12] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:36:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:36:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28713 [19:36:13] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 97006 [19:39:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:39:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:39:41] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56163.000000 [19:40:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:40:30] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:41:22] / on ortelius is WARNING: DISK WARNING - free space: / 4639 MB (15% inode=92%): [19:42:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 288515 MB (5% inode=64%): [19:44:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56312.000000 [19:46:03] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2135 [19:46:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:46:32] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2140.000000 [19:46:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:52:04] Platonides * Re: [Toolserver-l] Maintenance: Rebooting ortelius web server [19:54:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64499 MB (10% inode=99%): [20:00:03] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:03:49] Platonides: You could have said a word here as well :-). [20:06:23] aliasd on yarrow is CRITICAL: Connection refused [20:21:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:24:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:26:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:26:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 349538.000000 [20:27:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:27:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1126869.000000 [20:28:22] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:29:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:29:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:29:22] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:29:31] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:29:32] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:29:32] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:29:32] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:29:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:29:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:29:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2301323.000000 [20:29:42] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [20:29:42] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:29:42] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [20:30:01] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:02] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:03] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:23] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:30:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:30:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:30:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:32] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:32] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:32] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:33] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:42] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:30:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1412482.000000 [20:30:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:30:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:30:51] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 507467.000000 [20:31:02] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:02] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:02] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:02] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:31:03] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:31:04] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:31:04] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:31:11] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:31:11] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 695441.000000 [20:31:22] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:31:22] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:31:51] SSH on mayapple is CRITICAL: Server answer: [20:31:52] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:32:22] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:32:22] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:32:22] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:32:32] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:33:01] / on thyme is UNKNOWN: NRPE: Unable to read output [20:33:01] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:01] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:02] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1306636.000000 [20:33:21] aliasd on yarrow is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [20:34:35] §!"§//(§!"/! I had a query running since the early morning, was so happy that it didn't got killed, and now: "ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query". Fuck! [20:34:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:35:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:35:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:36:22] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25231 [20:36:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92127 [20:37:12] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:37:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:37:32] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:38:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:39:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:39:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:39:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58023.000000 [20:40:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:40:32] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:41:22] / on ortelius is WARNING: DISK WARNING - free space: / 4332 MB (14% inode=92%): [20:43:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 288225 MB (5% inode=64%): [20:43:42] Load avg. on nightshade is WARNING: WARNING - load average: 23.73, 18.32, 10.61 [20:44:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58205.000000 [20:46:02] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2926 [20:46:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:46:42] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2933.000000 [20:46:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:46:42] Load avg. on nightshade is OK: OK - load average: 5.44, 13.77, 10.35 [20:55:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64266 MB (10% inode=99%): [21:00:03] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:21:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:24:42] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:26:23] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:26:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 349902.000000 [21:27:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:28:22] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:28:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1130529.000000 [21:29:31] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:29:43] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:29:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2304392.000000 [21:29:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:29:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:29:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:29:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [21:29:44] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:29:44] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:29:45] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [21:30:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:03] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:03] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:04] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:23] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:30:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:30:23] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:30:31] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:30:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:31] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:42] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:42] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:42] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:43] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:43] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:30:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1415946.000000 [21:30:44] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:31:02] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:31:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:31:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:31:04] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:31:05] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:31:11] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:31:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 694869.000000 [21:31:22] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:31:51] SSH on mayapple is CRITICAL: Server answer: [21:31:51] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:31:51] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506991.000000 [21:32:22] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:32:22] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:32:22] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:32:22] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:32:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:33:02] / on thyme is UNKNOWN: NRPE: Unable to read output [21:33:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1309574.000000 [21:33:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:03] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:35:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:36:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:37:12] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:37:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:37:23] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87646 [21:37:23] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28500 [21:37:31] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:38:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:39:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:39:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:39:42] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60023.000000 [21:41:11] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:41:30] / on ortelius is WARNING: DISK WARNING - free space: / 4023 MB (13% inode=92%): [21:43:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 287946 MB (5% inode=64%): [21:43:12] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:43:32] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:43:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [21:43:51] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [21:44:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3602 [21:44:23] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60215.000000 [21:46:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:46:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:56:50] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63933 MB (10% inode=99%): [22:00:32] / on ortelius is OK: DISK OK - free space: / 7806 MB (26% inode=92%): [22:03:03] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:24:51] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:26:23] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:26:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 350582.000000 [22:27:31] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:28:22] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:28:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1134129.000000 [22:29:32] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:29:52] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:29:52] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [22:29:52] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [22:29:52] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:02] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:02] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:03] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:03] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:03] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:23] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:30:31] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:30:31] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:30:31] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:30:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:32] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2307671.000000 [22:30:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:42] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:42] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:30:51] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:51] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:52] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:30:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1419456.000000 [22:30:52] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:31:02] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:31:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:04] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:04] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:31:04] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:31:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:31:05] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:31:22] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:31:42] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:31:51] SSH on mayapple is CRITICAL: Server answer: [22:31:51] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:32:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506280.000000 [22:32:12] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:32:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 693829.000000 [22:32:21] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:32:22] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:32:22] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:32:31] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:32:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:33:03] / on thyme is UNKNOWN: NRPE: Unable to read output [22:33:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1311709.000000 [22:33:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:03] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:35:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:36:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:37:11] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:37:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:37:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82745 [22:37:22] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30640 [22:39:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:39:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:40:41] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62177.000000 [22:42:11] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:43:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 287732 MB (5% inode=64%): [22:44:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4192 [22:44:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62278.000000 [22:44:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4204.000000 [22:46:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:46:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:51:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:56:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63693 MB (10% inode=99%): [23:03:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:24:51] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:26:23] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:26:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 351926.000000 [23:27:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:28:22] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:28:41] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1137734.000000 [23:29:31] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:29:51] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [23:29:51] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:29:51] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [23:29:51] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:03] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:03] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:04] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:22] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:30:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:32] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:30:32] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:30:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:30:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2311050.000000 [23:30:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:51] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1422949.000000 [23:30:52] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:30:52] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:52] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:53] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:53] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:30:54] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:31:03] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:31:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:04] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:22] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:31:42] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:31:51] SSH on mayapple is CRITICAL: Server answer: [23:31:51] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:32:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:32:03] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:32:03] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:32:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:32:12] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:32:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 694310.000000 [23:32:22] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:32:23] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:32:23] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:32:52] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 505089.000000 [23:33:02] / on thyme is UNKNOWN: NRPE: Unable to read output [23:33:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1313749.000000 [23:33:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:33:02] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:33:03] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:33:31] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:33:42] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [23:35:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:36:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:36:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:37:11] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:37:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:37:23] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78369 [23:37:23] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32036 [23:39:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:39:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:40:41] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63936.000000 [23:42:11] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:44:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4996 [23:44:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 286807 MB (5% inode=64%): [23:44:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5014.000000 [23:45:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64111.000000 [23:46:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:46:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:51:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:56:18] @replag [23:56:19] Merlissimo: s1-rr-a: 21h 15m 48s [-0.02 s/s]; s1-rr-a-wd: 1w 4d 10h 55m 35s [+0.32 s/s]; s1-user: 1h 25m 57s [+0.03 s/s]; s1-user-c: 1w 6d 4h 30m 0s [+1.00 s/s]; s1-user-wd: 2w 2d 11h 40m 29s [+0.86 s/s]; s2-rr: 9h 3m 4s [-0.59 s/s]; s2-user: 9h 3m 4s [-0.59 s/s]; s2-user-c: error [23:56:20] Merlissimo: s2-user-wd: 2w 1d 5h 11m 7s [+0.12 s/s]; s3-user: 2m 46s [+0.00 s/s]; s3-user-wd: 1w 1d 6h 19m 27s [-0.39 s/s]; s4-user-wd: 3w 5d 18h 21m 37s [+0.88 s/s]; s5-rr-a: 17h 49m 47s [+0.59 s/s]; s5-rr-a-wd: 4d 1h 45m 49s [-0.04 s/s]; s5-user: 17h 49m 47s [-0.19 s/s]; s6-user-wd: 1w 1d 45m 5s [-0.33 s/s] [23:56:21] Merlissimo: s7-user-wd: 5d 20h 16m 8s [-0.22 s/s] [23:56:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63387 MB (10% inode=99%):