[00:03:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [00:07:09] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:08:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:09:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70028 MB (11% inode=99%): [00:09:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338233 MB (6% inode=63%): [00:09:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:09:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:09:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:10:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 553149.000000 [00:10:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:10:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 631504.000000 [00:11:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:11:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:11:48] aliasd on yarrow is CRITICAL: Connection refused [00:12:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 365105.000000 [00:12:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:12:58] / on thyme is UNKNOWN: NRPE: Unable to read output [00:13:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:13:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:13:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:14:27] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:14:28] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:14:48] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:07] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:28] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:38] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:16:47] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:48] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:16:48] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:48] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:58] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:16:58] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:58] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:59] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:17:08] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:17:18] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:17:19] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:17:39] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:17:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:17:58] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:17:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:08] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:18:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:18:08] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:18:08] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:18:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:19:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:19:28] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:19:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:20:28] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:20:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:20:38] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:28:48] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7233.000000 [00:28:48] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7233.000000 [00:31:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:33:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555768.000000 [00:34:30] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:37:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9899.000000 [00:40:59] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9897 [00:47:38] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [01:04:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:07:09] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:08:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:09:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69820 MB (11% inode=99%): [01:09:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338763 MB (6% inode=63%): [01:09:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:09:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:09:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:11:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556809.000000 [01:11:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 635165.000000 [01:11:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:11:48] aliasd on yarrow is CRITICAL: Connection refused [01:12:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368704.000000 [01:12:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:12:58] / on thyme is UNKNOWN: NRPE: Unable to read output [01:13:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:13:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:14:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:14:28] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:14:28] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:14:47] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:08] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:27] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:37] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:16:48] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:48] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:16:48] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:48] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:58] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:16:58] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:58] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:58] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:17:09] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:17:18] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:17:39] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:17:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:17:59] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:17:59] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:17] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:19:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:19:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:19:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:19:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:19:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:19:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:19:28] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:19:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:20:28] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:20:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:20:38] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:22:02] ... [01:28:48] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9290.000000 [01:28:49] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9124.000000 [01:31:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:34:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 558868.000000 [01:35:29] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:38:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:38:49] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3397.000000 [01:40:48] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3348.000000 [01:41:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7660.000000 [01:42:00] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7662 [01:43:49] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1690.000000 [01:45:49] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1667.000000 [01:47:38] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [02:04:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [02:07:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:08:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:09:17] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67741 MB (11% inode=99%): [02:09:38] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2057.000000 [02:09:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:10:08] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2079 [02:10:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338704 MB (6% inode=63%): [02:10:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:10:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:11:08] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 558573.000000 [02:11:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 636907.000000 [02:11:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:11:48] aliasd on yarrow is CRITICAL: Connection refused [02:11:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:12:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 369638.000000 [02:12:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:12:58] / on thyme is UNKNOWN: NRPE: Unable to read output [02:13:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:13:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:14:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:14:28] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:14:29] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:14:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:08] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:09] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:30] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:16:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:49] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:16:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:59] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:17:00] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:17:00] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:17:00] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:17:09] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:17:19] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:17:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:17:59] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:18:00] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:18:00] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:19] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:18:39] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:18:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:19:28] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:19:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:20:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:20:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:20:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:20:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:20:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:20:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:20:39] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:20:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:20:39] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:32:19] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66811 MB (10% inode=99%): [02:33:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71069 MB (11% inode=99%): [02:34:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 559150.000000 [02:35:29] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:38:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:41:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8415.000000 [02:41:59] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8415 [02:47:39] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [02:53:39] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [02:54:09] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3627 [03:01:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:03:39] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2106 [03:03:59] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2124.000000 [03:04:09] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3531 [03:04:38] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3238.000000 [03:04:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [03:07:09] MySQL slave on daphne is OK: Uptime: 847410 Threads: 32 Questions: 319882077 Slow queries: 44127 Opens: 24329 Flush tables: 1 Open tables: 1452 Queries per second avg: 377.482 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1699 [03:07:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:07:40] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1340.000000 [03:08:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:10:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338194 MB (6% inode=63%): [03:10:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:10:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:10:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:11:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 559392.000000 [03:11:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 637818.000000 [03:11:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:11:48] aliasd on yarrow is CRITICAL: Connection refused [03:11:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:19] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368090.000000 [03:12:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:12:58] / on thyme is UNKNOWN: NRPE: Unable to read output [03:13:29] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:14:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:14:28] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:14:29] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:14:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:14:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:08] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:28] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:37] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:17:47] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:48] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:17:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:58] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:17:59] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:59] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:59] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:18:08] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:18:18] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:18:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:18:58] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:18:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:18:58] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:19:18] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:19:37] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:19:38] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:20:28] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:20:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:20:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:20:38] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:21:08] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:21:08] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:21:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:21:08] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:21:08] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:21:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:21:39] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:31:50] Where is the centralauth db stored? [03:31:53] Susan: ^ [03:32:56] sql-s3, I think. [03:33:05] s7. [03:33:15] $ sql centralauth_p; [03:33:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69639 MB (11% inode=99%): [03:33:24] mysql> select @@hostname\G [03:33:32] *************************** 1. row *************************** [03:33:32] @@hostname: z-dat-s7-a [03:33:34] thanks [03:34:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 559620.000000 [03:35:38] MySQL slave on z-dat-s5-b is OK: Uptime: 108221 Threads: 7 Questions: 120422118 Slow queries: 679 Opens: 95234 Flush tables: 1 Open tables: 3878 Queries per second avg: 1112.742 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [03:36:00] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1731.000000 [03:36:28] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:38:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:43:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8622.000000 [03:43:00] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8620 [03:47:39] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [04:01:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:04:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [04:08:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:08:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:10:29] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338548 MB (6% inode=63%): [04:10:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:10:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:10:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:11:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560691.000000 [04:11:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 638791.000000 [04:11:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:11:48] aliasd on yarrow is CRITICAL: Connection refused [04:11:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:12:19] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 366654.000000 [04:12:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:13:00] / on thyme is UNKNOWN: NRPE: Unable to read output [04:13:29] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:14:10] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:14:29] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:14:29] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:15:08] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:15:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:29] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:17:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:49] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:17:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:59] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:17:59] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:59] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:59] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:18:08] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:18:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:18:19] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:18:59] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:18:59] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:19:00] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:19:19] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:19:39] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:19:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:20:28] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:20:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:21:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:21:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:21:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:21:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:21:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:21:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:21:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:21:38] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:21:38] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:33:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69596 MB (11% inode=99%): [04:35:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560520.000000 [04:36:28] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:39:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:44:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9422.000000 [04:44:00] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9423 [04:47:39] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [05:01:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:05:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [05:08:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:08:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:10:29] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338502 MB (6% inode=63%): [05:10:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:10:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:11:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 561936.000000 [05:11:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:11:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:11:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:11:49] aliasd on yarrow is CRITICAL: Connection refused [05:11:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:12:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 640312.000000 [05:13:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 361715.000000 [05:13:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:13:29] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:13:59] / on thyme is UNKNOWN: NRPE: Unable to read output [05:14:28] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:15:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:15:29] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:15:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:15:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:30] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:17:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:49] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:17:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:59] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:17:59] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:59] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:18:00] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:18:09] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:18:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:18:18] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:19:19] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:19:39] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:19:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:19:59] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:19:59] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:19:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:20:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:21:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:21:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:21:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:21:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:21:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:21:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:21:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:21:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:21:39] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:34:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69516 MB (11% inode=99%): [05:36:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 561260.000000 [05:36:29] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:40:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:44:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10946.000000 [05:44:59] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10948 [05:47:39] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [06:04:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:08:09] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:08:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:10:29] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338403 MB (6% inode=63%): [06:10:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:10:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:11:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:12:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 563357.000000 [06:12:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:12:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 641100.000000 [06:12:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:12:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:12:49] aliasd on yarrow is CRITICAL: Connection refused [06:13:19] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354195.000000 [06:13:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:13:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:13:59] / on thyme is UNKNOWN: NRPE: Unable to read output [06:14:29] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:15:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:09] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:15:28] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:15:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:15:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:17:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:17:09] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:17:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:17:49] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:17:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:17:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:17:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:17:59] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:18:00] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:18:00] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:18:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:18:19] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:18:29] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:18:59] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:19:09] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:19:19] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:19:39] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:19:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:20:58] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:20:59] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:20:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:21:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:21:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:21:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:21:39] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:21:39] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:22:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:22:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:22:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:22:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:22:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:31:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:34:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69417 MB (11% inode=99%): [06:36:29] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:37:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 561852.000000 [06:40:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:44:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9198.000000 [06:44:59] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9193 [06:47:39] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [07:05:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [07:08:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:09:09] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:10:30] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338293 MB (6% inode=63%): [07:10:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:10:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:11:58] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:12:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:12:49] aliasd on yarrow is CRITICAL: Connection refused [07:13:08] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 564723.000000 [07:13:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:13:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 641581.000000 [07:13:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347920.000000 [07:13:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:14:00] / on thyme is UNKNOWN: NRPE: Unable to read output [07:14:29] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [07:14:29] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:15:08] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:16:28] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [07:16:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:16:48] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:17:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [07:17:49] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [07:17:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:17:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:17:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:18:00] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:18:00] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:18:00] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:18:09] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:18:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:18:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:18:19] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:28] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:19:58] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [07:20:08] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:20:18] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:20:38] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [07:20:38] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:21:00] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [07:21:00] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:00] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:21:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:21:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:21:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:21:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:21:39] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [07:22:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:22:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:22:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:22:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:22:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:34:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69226 MB (11% inode=99%): [07:37:29] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:38:08] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 562024.000000 [07:40:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:45:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4530.000000 [07:45:00] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4530 [07:47:39] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [08:01:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:04:00] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3594 [08:04:59] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3510.000000 [08:05:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [08:08:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:09:09] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:10:29] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 338193 MB (6% inode=63%): [08:10:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:10:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:11:58] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:12:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:12:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:12:49] aliasd on yarrow is CRITICAL: Connection refused [08:13:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 565978.000000 [08:13:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:13:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 642394.000000 [08:13:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 343785.000000 [08:13:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:14:58] / on thyme is UNKNOWN: NRPE: Unable to read output [08:15:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:15:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:28] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [08:15:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:16:29] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [08:16:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:16:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:17:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [08:17:49] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [08:17:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:17:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:17:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:18:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:18:09] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:18:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:18:19] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:18:29] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:18:59] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:19:00] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:19:00] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:20:00] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [08:20:08] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:20:18] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:20:38] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [08:20:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:21:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:21:39] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:21:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:21:39] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [08:21:59] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [08:21:59] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:22:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:22:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:22:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:22:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:22:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:22:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:24:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:24:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 343089.000000 [08:24:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:24:04] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:24:04] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:24:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:24:04] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1914 [08:24:05] / on thyme is UNKNOWN: NRPE: Unable to read output [08:24:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:24:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:24:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:24:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:24:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:24:14] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:34:03] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 29849 MB (5% inode=99%): [08:34:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 550589.000000 [08:35:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69006 MB (11% inode=99%): [08:38:12] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:38:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 562532.000000 [08:39:22] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5353 [08:39:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 558458.000000 [08:41:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:48:23] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [08:53:42] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [08:54:02] MySQL slave on rosemary is OK: Uptime: 1531784 Threads: 19 Questions: 1343288913 Slow queries: 464510 Opens: 70112 Flush tables: 1 Open tables: 3467 Queries per second avg: 876.944 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1774 [09:23:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:23:02] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [09:23:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:23:13] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:23:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:23:13] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [09:23:13] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [09:23:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 337975 MB (6% inode=63%): [09:23:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:23:15] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:15] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:16] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [09:23:23] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:23:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:23:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:23] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [09:23:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:23:34] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:34] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 567792.000000 [09:23:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:34] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [09:23:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:23:35] aliasd on yarrow is CRITICAL: Connection refused [09:23:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:23:43] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [09:23:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:54] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:54] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:54] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:23:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 643937.000000 [09:23:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:23:55] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:24:02] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:24:02] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:24:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:25:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 339925.000000 [09:25:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:25:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:25:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:25:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:25:03] / on thyme is UNKNOWN: NRPE: Unable to read output [09:25:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:25:12] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:25:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:25:14] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:25:14] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:25:14] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:34:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 551503.000000 [09:35:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68780 MB (11% inode=99%): [09:38:14] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:38:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 563630.000000 [09:39:22] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6336 [09:39:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560917.000000 [09:41:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:48:23] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [10:09:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 22134 MB (2% inode=99%): [10:17:40] [[Special:Log/newusers]] create 10 * I-pon * (New user account) [10:23:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:23:02] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [10:23:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:23:12] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:23:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [10:23:22] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [10:23:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:23:33] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:23:43] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [10:23:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:23:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:23:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:23:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:03] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:24:13] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [10:24:13] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [10:24:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 336946 MB (6% inode=63%): [10:24:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:24:14] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:14] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:15] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:22] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [10:24:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:24:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:32] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 568777.000000 [10:24:32] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:32] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [10:24:32] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:24:33] aliasd on yarrow is CRITICAL: Connection refused [10:24:33] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:34] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:34] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:24:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:52] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:52] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:52] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:24:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 646639.000000 [10:24:53] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:24:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 338872.000000 [10:26:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:26:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:26:03] / on thyme is UNKNOWN: NRPE: Unable to read output [10:26:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:26:14] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:34:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 552376.000000 [10:35:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68568 MB (11% inode=99%): [10:39:12] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:39:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 564962.000000 [10:39:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 563468.000000 [10:40:22] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7550 [10:41:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:48:23] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [10:54:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:05:02] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1876 [11:05:42] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1873.000000 [11:23:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:23:03] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [11:23:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:23:13] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:23:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [11:23:23] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [11:23:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:23:33] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:23:42] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [11:23:42] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:23:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:23:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:23:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:02] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:24:02] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:24:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:24:12] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [11:24:12] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [11:24:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 337015 MB (6% inode=63%): [11:24:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:24:13] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:24:13] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:24:14] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:24:32] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:24:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 570230.000000 [11:24:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:33] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [11:24:33] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:24:33] aliasd on yarrow is CRITICAL: Connection refused [11:24:33] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:24:34] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:24:34] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:25:23] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [11:25:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:25:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:25:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:25:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:25:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:52] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:25:52] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:25:53] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:25:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 649627.000000 [11:25:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:25:53] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:26:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 338440.000000 [11:26:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:26:02] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:26:02] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:26:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:26:03] / on thyme is UNKNOWN: NRPE: Unable to read output [11:27:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:27:14] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:34:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 553129.000000 [11:35:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68328 MB (11% inode=99%): [11:39:13] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:39:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 565797.000000 [11:39:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 566860.000000 [11:41:22] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10233 [11:42:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:48:22] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [11:54:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:05:03] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3145 [12:05:42] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3164.000000 [12:23:03] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:23:03] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [12:23:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [12:24:02] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:24:02] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:24:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:24:12] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:24:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:24:13] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [12:24:13] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [12:24:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 336819 MB (6% inode=63%): [12:24:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:24:14] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:14] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:15] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:22] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [12:24:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:24:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:34] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 572361.000000 [12:24:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:34] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [12:24:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:24:34] aliasd on yarrow is CRITICAL: Connection refused [12:24:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:35] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:24:42] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [12:24:42] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:42] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:24:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:52] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:25:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:25:53] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:25:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 653047.000000 [12:25:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:25:53] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:26:22] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [12:26:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:26:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:26:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:26:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 337797.000000 [12:27:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:27:02] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:27:03] / on thyme is UNKNOWN: NRPE: Unable to read output [12:27:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:12] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:27:42] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2110.000000 [12:34:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3604 [12:34:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [12:34:52] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 553750.000000 [12:35:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68067 MB (11% inode=99%): [12:39:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 567297.000000 [12:39:52] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 569558.000000 [12:40:13] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:41:22] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11184 [12:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:48:22] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [12:54:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:58:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3639.000000 [13:07:36] wtf [13:07:43] i'm getting 404s for everything [13:09:41] keep refreshing [13:23:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:23:03] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [13:23:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [13:24:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:24:03] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:24:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:24:14] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:24:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:24:14] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [13:24:14] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [13:24:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 336520 MB (6% inode=63%): [13:24:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:24:15] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:24:15] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:24:16] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:24:23] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [13:24:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:24:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:24:42] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [13:24:42] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:24:42] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:24:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:24:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:32] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 574286.000000 [13:25:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:34] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [13:25:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:25:34] aliasd on yarrow is CRITICAL: Connection refused [13:25:34] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:25:34] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:25:35] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:25:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:54] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:25:54] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:25:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:25:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:25:54] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:26:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 656562.000000 [13:26:22] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [13:26:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:26:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:26:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:26:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 337054.000000 [13:27:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:27:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:27:03] / on thyme is UNKNOWN: NRPE: Unable to read output [13:27:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:14] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:14] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:27:14] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:34:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5657 [13:34:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5664.000000 [13:35:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 554840.000000 [13:36:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67835 MB (11% inode=99%): [13:39:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 569704.000000 [13:40:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 572404.000000 [13:40:13] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:41:32] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12229 [13:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:44:30] Hello all [13:47:02] DaB|Balkon: any idea why we are still getting random 404's ? [13:48:22] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [13:54:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:56:06] Betacommand: Better ask nosy, she switched the webserver-config. [13:58:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6969.000000 [14:23:03] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:23:03] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [14:23:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [14:24:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:24:03] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:24:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:24:43] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [14:24:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:24:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:24:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:24:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:13] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:25:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:25:13] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [14:25:13] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [14:25:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 336370 MB (6% inode=63%): [14:25:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:25:14] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:25:15] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:25:15] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:25:22] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [14:25:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:25:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:25:33] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:25:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:33] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [14:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 576371.000000 [14:25:33] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:25:34] aliasd on yarrow is CRITICAL: Connection refused [14:25:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:25:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:25:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:26:22] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [14:26:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:26:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:26:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:26:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:26:32] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:26:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:26:52] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:26:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 660111.000000 [14:26:52] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:26:52] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 335462.000000 [14:27:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:27:02] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:02] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:27:03] / on thyme is UNKNOWN: NRPE: Unable to read output [14:27:12] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:14] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:14] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:27:14] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:28:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:34:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5662.000000 [14:35:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555377.000000 [14:35:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5671 [14:36:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67616 MB (11% inode=99%): [14:39:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 570974.000000 [14:40:13] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:40:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 574823.000000 [14:41:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11236 [14:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:49:23] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [14:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:59:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8235.000000 [15:23:02] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [15:23:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:23:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [15:24:03] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:24:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:24:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:24:43] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [15:24:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:24:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:24:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:24:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:12] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:25:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:25:14] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [15:25:14] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [15:25:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 336185 MB (6% inode=63%): [15:25:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:25:14] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:25:14] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:25:15] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:25:23] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [15:25:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:25:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:25:34] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:25:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:34] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [15:25:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 578736.000000 [15:25:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:25:34] aliasd on yarrow is CRITICAL: Connection refused [15:25:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:25:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:25:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:26:22] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [15:26:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:26:22] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:26:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:26:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:26:32] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:26:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:26:52] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:26:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 662624.000000 [15:26:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:26:53] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:27:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 333967.000000 [15:27:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:27:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:27:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:27:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:27:03] / on thyme is UNKNOWN: NRPE: Unable to read output [15:28:12] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:12] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:12] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:12] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:28:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:35:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6428.000000 [15:36:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555367.000000 [15:36:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6434 [15:37:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67438 MB (11% inode=99%): [15:39:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 573160.000000 [15:40:13] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:40:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 577255.000000 [15:41:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11222 [15:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:50:23] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [15:59:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4575.000000 [16:23:02] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [16:23:03] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:23:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [16:24:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:24:03] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:24:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:24:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:24:42] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [16:24:42] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:24:42] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:24:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:24:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:12] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:25:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:25:13] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [16:25:13] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [16:25:23] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [16:25:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:26:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 335799 MB (6% inode=63%): [16:26:12] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:26:12] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:12] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:12] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:26:32] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:33] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [16:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 581348.000000 [16:26:33] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:34] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:26:35] aliasd on yarrow is CRITICAL: Connection refused [16:26:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:53] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:26:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 665287.000000 [16:26:53] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:26:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:27:22] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [16:27:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:27:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:27:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:27:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 331861.000000 [16:28:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:28:04] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:04] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:28:05] / on thyme is UNKNOWN: NRPE: Unable to read output [16:28:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:28:14] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:35:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7865.000000 [16:36:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555194.000000 [16:36:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7889 [16:37:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67187 MB (11% inode=99%): [16:40:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 574940.000000 [16:41:12] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:41:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 579679.000000 [16:42:32] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10698 [16:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:44:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3586.000000 [16:50:07] closedmouth: Try to not use HTTPS. [16:50:23] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [17:06:42] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [17:09:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67091 MB (10% inode=99%): [17:23:02] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [17:23:03] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:23:23] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [17:24:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:24:02] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:24:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:24:42] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [17:24:42] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:24:42] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:24:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:24:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:25:13] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:25:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:25:13] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [17:25:13] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [17:25:33] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [17:25:42] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:26:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 335456 MB (6% inode=63%): [17:26:12] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:26:12] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:12] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:12] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:26:32] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:33] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [17:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 584301.000000 [17:26:33] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:34] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:26:35] aliasd on yarrow is CRITICAL: Connection refused [17:26:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:53] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:26:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 667701.000000 [17:26:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:26:53] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:27:23] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [17:27:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:27:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:27:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:27:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 329088.000000 [17:28:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:28:03] / on thyme is UNKNOWN: NRPE: Unable to read output [17:28:12] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:28:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:04] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:04] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:29:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:35:07] OK What's broken folks? [17:35:15] https://toolserver.org/~magnus/catscan_rewrite.php [17:35:24] Was working up until about 10 mins ago [17:36:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9564.000000 [17:37:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555205.000000 [17:37:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9572 [17:40:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 576403.000000 [17:41:13] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:41:23] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 582190.000000 [17:42:25] Try without https. [17:42:32] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10248 [17:42:44] Or http://ortelius.toolserver.org/~magnus/catscan_rewrite.php [17:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:50:22] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [18:09:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66810 MB (10% inode=99%): [18:23:03] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [18:23:03] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:24:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:24:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:24:03] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:24:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [18:24:43] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [18:24:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:24:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:24:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:24:43] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:42] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:26:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:26:12] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:26:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:26:12] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [18:26:13] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [18:26:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 335235 MB (6% inode=63%): [18:26:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:26:14] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:14] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:15] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:32] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [18:26:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:26:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:33] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:33] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [18:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 587668.000000 [18:26:34] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:26:35] aliasd on yarrow is CRITICAL: Connection refused [18:26:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:53] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:26:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 669617.000000 [18:26:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:26:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 325818.000000 [18:28:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:22] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [18:28:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:28:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:28:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:29:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:29:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:29:03] / on thyme is UNKNOWN: NRPE: Unable to read output [18:37:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555210.000000 [18:37:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10837 [18:37:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10838.000000 [18:40:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 578210.000000 [18:41:13] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:41:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 584781.000000 [18:42:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9332 [18:43:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:51:23] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [18:59:04] Marlen Caemmerer * Re: [Toolserver-l] Failed Maintenance - was - Re: IP Renumbering of the complete Toolserver cluster (fwd) [19:10:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66642 MB (10% inode=99%): [19:23:03] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:23:13] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:24:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:24:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:24:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:24:22] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:24:53] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:42] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:25:42] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:25:42] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:25:42] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:26:33] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:26:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:26:33] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:26:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:26:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:33] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 590407.000000 [19:26:34] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:26:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:26:35] aliasd on yarrow is CRITICAL: Connection refused [19:26:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:26:42] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:27:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:27:12] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:27:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:27:12] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:27:12] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:27:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 335026 MB (6% inode=63%): [19:27:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:27:14] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:27:14] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:27:15] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:27:52] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:27:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 670646.000000 [19:27:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:27:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:27:54] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:27:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 321423.000000 [19:28:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:22] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:28:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:28:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:28:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:29:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:29:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:29:03] / on thyme is UNKNOWN: NRPE: Unable to read output [19:37:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12447 [19:37:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555555.000000 [19:37:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12433.000000 [19:40:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 580374.000000 [19:41:13] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:42:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 587084.000000 [19:43:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:44:33] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [19:52:22] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [20:00:00] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 318967.000000 [20:00:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:00:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:00:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [20:00:00] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12552 [20:00:00] / on thyme is UNKNOWN: NRPE: Unable to read output [20:00:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:09] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:09] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:00:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:00] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66435 MB (10% inode=99%): [20:34:59] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 29106 MB (5% inode=99%): [20:59:18] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:59:39] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:59:50] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:59:50] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:59:50] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:59:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 581982.000000 [20:59:50] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:59:51] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 671521.000000 [20:59:51] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:59:52] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:00:00] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 314190.000000 [21:00:00] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 590865.000000 [21:00:00] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556312.000000 [21:00:00] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:00:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:00:01] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:02] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:02] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:00:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [21:00:03] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:00:04] / on thyme is UNKNOWN: NRPE: Unable to read output [21:00:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12854 [21:00:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:09] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:00:09] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:00:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:00:21] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:00:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:00:29] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:29] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:29] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 593232.000000 [21:00:30] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:30] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:00:30] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:00:31] aliasd on yarrow is CRITICAL: Connection refused [21:00:31] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:32] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:00:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:43] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:00:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:00:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12857.000000 [21:00:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:11:59] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66074 MB (10% inode=99%): [21:54:00] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:55:59] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 28785 MB (5% inode=99%): [21:59:19] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:59:39] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:50] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:59:50] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:59:50] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 582379.000000 [21:59:50] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:59:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 672777.000000 [21:59:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:59:51] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:00:00] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 309280.000000 [22:00:00] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556302.000000 [22:00:00] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:00:01] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 593892.000000 [22:00:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:00:01] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:01] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:02] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:00:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [22:00:03] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:00:03] / on thyme is UNKNOWN: NRPE: Unable to read output [22:00:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13290 [22:00:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:09] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:00:09] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:00:10] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:00:39] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:00:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:00:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13294.000000 [22:00:39] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:39] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:40] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:19] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9775 [22:01:20] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:20] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:20] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:01:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:01:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:30] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 595060.000000 [22:01:30] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:30] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:01:30] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:01:31] aliasd on yarrow is CRITICAL: Connection refused [22:01:31] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:32] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:01:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:02:30] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [22:02:30] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [22:10:19] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 21844 MB (2% inode=99%): [22:12:00] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65926 MB (10% inode=99%): [22:17:11] nacht ts [22:24:29] MySQL slave on z-dat-s3-a is OK: Uptime: 1714 Threads: 8 Questions: 2694 Slow queries: 0 Opens: 112 Flush tables: 1 Open tables: 96 Queries per second avg: 1.571 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1734 [22:24:30] MySQL on z-dat-s3-a is OK: Uptime: 1714 Threads: 8 Questions: 2696 Slow queries: 0 Opens: 112 Flush tables: 1 Open tables: 96 Queries per second avg: 1.572 [22:26:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 582583.000000 [22:26:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:26:20] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556025.000000 [22:26:20] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 595051.000000 [22:26:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:26:21] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 672988.000000 [22:26:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:26:22] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:26:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:26:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 307527.000000 [22:26:24] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:26:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:26:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:26:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:28:45] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1967 [22:50:44] MySQL slave on z-dat-s3-a is OK: Uptime: 3293 Threads: 15 Questions: 587151 Slow queries: 102 Opens: 10850 Flush tables: 1 Open tables: 10833 Queries per second avg: 178.302 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1679 [22:59:04] Hi whats up? [22:59:24] I'm getting a large number of 404 errors [23:25:25] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:25:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:25:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:25:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:25:35] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:25:35] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:25:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 333206 MB (6% inode=63%): [23:25:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:25:37] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:37] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:44] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:25:44] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:25:44] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:25:44] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [23:25:44] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7450 [23:25:45] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:45] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:46] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:46] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:25:54] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:25:54] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:54] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 597751.000000 [23:25:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:25:55] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:25:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:25:56] aliasd on yarrow is CRITICAL: Connection refused [23:25:56] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:57] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:25:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:26:14] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555163.000000 [23:26:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 583752.000000 [23:26:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 597404.000000 [23:26:15] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:26:15] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:26:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:15] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:26:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 673083.000000 [23:26:16] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:26:17] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:26:35] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:26:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:26:35] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:26:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:04] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:27:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11653.000000 [23:27:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:27:04] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:27:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:27:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:27:05] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 303772.000000 [23:27:24] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65726 MB (10% inode=99%): [23:27:24] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:27:25] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:27:25] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:27:25] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:27:26] / on thyme is UNKNOWN: NRPE: Unable to read output [23:27:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11652 [23:27:34] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:27:34] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:27:34] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:27:34] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:38:38] [[~dispenser/view/Checklinks]] ! 10https://wiki.toolserver.org/w/index.php?diff=7905&oldid=7120&rcid=21758 * 82.154.34.166 * (+0) (just a typo) [23:56:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL