[00:25:24] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:25:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:25:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:25:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:25:36] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:25:36] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:25:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 329836 MB (6% inode=63%): [00:25:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:25:36] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:37] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:37] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:44] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:25:44] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:25:44] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:25:44] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [00:25:55] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:25:55] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:55] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 599880.000000 [00:25:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:55] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:25:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:25:56] aliasd on yarrow is CRITICAL: Connection refused [00:25:57] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:57] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:25:58] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:26:15] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555028.000000 [00:26:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 583886.000000 [00:26:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 599461.000000 [00:26:15] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:15] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:26:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 675684.000000 [00:26:17] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:18] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:35] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:35] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:44] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5529 [00:26:44] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:44] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:44] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:45] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:27:13] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [00:27:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:27:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11813.000000 [00:27:13] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:13] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:13] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:25] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:27:25] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65594 MB (10% inode=99%): [00:27:25] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 300791.000000 [00:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:27:25] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:25] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:26] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:27:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:27:26] / on thyme is UNKNOWN: NRPE: Unable to read output [00:27:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11840 [00:27:35] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:27:35] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:35] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:56:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:25:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:25:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:25:44] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:25:45] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:25:45] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:25:45] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [01:25:55] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:25:55] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:25:55] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:25:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 601844.000000 [01:25:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:55] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:25:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:25:56] aliasd on yarrow is CRITICAL: Connection refused [01:26:25] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:26:35] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:35] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:26:36] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:26:36] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:26:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332657 MB (6% inode=63%): [01:26:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:26:37] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:38] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:44] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5830 [01:26:44] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:45] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:45] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:45] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:26:55] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:55] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:27:05] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [01:27:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:27:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13689.000000 [01:27:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:05] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:06] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:06] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:14] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 554058.000000 [01:27:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 601817.000000 [01:27:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 584633.000000 [01:27:14] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:14] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:15] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:27:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 678426.000000 [01:27:16] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:17] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:27] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65324 MB (10% inode=99%): [01:27:27] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 297394.000000 [01:27:27] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:27:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:27:27] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:27] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:27] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:27:34] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:27:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:35] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:28:25] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:28:25] / on thyme is UNKNOWN: NRPE: Unable to read output [01:28:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13680 [01:31:01] TS seems kinda patchy right now...I'm getting 404s? [01:31:44] Maple__: Works fine here [01:31:51] https://toolserver.org/~cvn/ [01:31:53] https://toolserver.org/~krinkle/ [01:32:10] HTTP is down I see [01:32:11] different server [01:32:18] must be something on wolfsbane [01:32:29] http://wolfsbane.toolserver.org/~krinkle/ [01:32:33] Also works fine [01:32:44] ortelius is also fine: http://ortelius.toolserver.org/~krinkle/ [01:32:55] Then its an issue in the load balancer? [01:33:07] Maple__: Since when did you see this? I always use HTTPS so I didn't notice it myself earlier [01:33:53] I'm connecting to HTTPS and it's down for me. [01:33:53] https://toolserver.org/~tparis/count/index.php?name=Kevin+McE&lang=en&wiki=wikipedia [01:33:57] Not Found [01:33:58] The requested URL /~tparis/count/index.php was not found on this server. [01:34:03] it's kinda been patchy today for me [01:48:04] Load avg. on daphne is WARNING: WARNING - load average: 15.79, 15.32, 14.16 [01:56:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:57:05] Load avg. on daphne is OK: OK - load average: 14.23, 14.93, 14.55 [02:09:05] Load avg. on daphne is WARNING: WARNING - load average: 15.61, 15.16, 14.80 [02:16:13] I've been getting the same thing as Maple__, above, for several days. [02:17:41] http://toolserver.org/~magnus/ "The requested URL /~magnus/ was not found on this server." [02:17:56] "The requested URL /~snottywong/adminscore.html was not found on this server." [02:18:30] "The requested URL /~tparis/ was not found on this server." [02:19:08] http://toolserver.org/~pietrodn/sectionLinks.php "The requested URL /~pietrodn/sectionLinks.php was not found on this server." [02:19:17] just some examples of 404s [02:25:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:25:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:25:44] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:25:44] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:25:44] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:25:44] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [02:25:54] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:25:54] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:54] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:25:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 603644.000000 [02:25:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:55] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:25:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:25:56] aliasd on yarrow is CRITICAL: Connection refused [02:26:25] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:26:35] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:35] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:26:45] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4951 [02:26:45] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:45] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:45] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:45] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:27:05] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:27:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:27:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15533.000000 [02:27:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:05] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:06] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:06] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 585122.000000 [02:27:15] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 554659.000000 [02:27:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 604229.000000 [02:27:15] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:15] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:27:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 681196.000000 [02:27:17] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:18] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:27:35] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:27:35] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:27:35] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [02:27:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330572 MB (6% inode=63%): [02:27:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:27:37] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:37] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:38] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:39] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:39] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:54] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:54] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:28:25] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63227 MB (10% inode=99%): [02:28:25] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 294301.000000 [02:28:25] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:28:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:28:25] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:28:25] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:28:26] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:28:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:29:24] / on thyme is UNKNOWN: NRPE: Unable to read output [02:29:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15493 [02:31:50] Maple__: tucoxn: Report on the mailing list? [02:32:05] Load avg. on daphne is OK: OK - load average: 10.49, 13.40, 14.71 [02:45:04] Krinkle * Re: [Toolserver-l] Web server issues (was: Failed Maintenance) [02:56:35] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:25:45] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:25:45] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:25:45] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:25:45] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [03:25:54] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:25:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 606204.000000 [03:25:55] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:55] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:25:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:55] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:25:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:25:56] aliasd on yarrow is CRITICAL: Connection refused [03:26:25] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:26:36] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:36] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:26:36] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:27:05] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:27:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:27:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15215.000000 [03:27:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:05] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:06] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:06] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 585805.000000 [03:27:15] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555263.000000 [03:27:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 606431.000000 [03:27:15] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:15] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:27:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 683185.000000 [03:27:17] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:17] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:45] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3703 [03:27:45] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:45] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:45] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:45] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:27:55] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:27:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:27:55] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:25] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:28:25] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65046 MB (10% inode=99%): [03:28:25] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 291357.000000 [03:28:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:28:26] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:26] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:26] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:28:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:28:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:28:35] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:28:35] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:28:35] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [03:28:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331771 MB (6% inode=63%): [03:28:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:28:37] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:37] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:38] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:39] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:28:39] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:29:24] / on thyme is UNKNOWN: NRPE: Unable to read output [03:29:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15196 [04:25:45] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:25:46] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:25:46] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:25:46] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [04:25:46] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3437 [04:25:56] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:25:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 608365.000000 [04:25:56] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:56] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:25:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:56] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:25:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:25:57] aliasd on yarrow is CRITICAL: Connection refused [04:26:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:26:26] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:26:36] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:26:36] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:26:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:26:36] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:27:04] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:27:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:27:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16214.000000 [04:27:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:05] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 587751.000000 [04:27:15] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556146.000000 [04:27:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 608915.000000 [04:27:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:27:16] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:16] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:27:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 685740.000000 [04:27:17] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:18] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:45] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:46] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:46] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:46] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:27:56] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:56] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:27:56] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:28:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:28:05] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:28:25] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64932 MB (10% inode=99%): [04:28:25] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:28:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:28:25] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:28:25] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:28:26] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:28:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:28:34] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:28:34] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:28:35] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [04:28:35] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:28:35] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:28:35] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332421 MB (6% inode=63%): [04:28:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:28:36] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:28:36] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:29:26] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 285881.000000 [04:29:26] / on thyme is UNKNOWN: NRPE: Unable to read output [04:29:26] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16359 [04:29:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:29:36] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:29:36] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:29:36] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:44] MySQL slave on z-dat-s6-a is OK: Uptime: 505495 Threads: 13 Questions: 286612429 Slow queries: 31417 Opens: 778476 Flush tables: 1 Open tables: 3179 Queries per second avg: 566.993 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1797 [05:25:45] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:25:45] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:26:35] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:26:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:26:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:26:44] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:44] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:45] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:26:45] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [05:26:55] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:26:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 610454.000000 [05:26:55] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:55] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:26:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:56] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:26:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:26:56] aliasd on yarrow is CRITICAL: Connection refused [05:27:05] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:27:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:27:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17444.000000 [05:27:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:27:05] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:27:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 589371.000000 [05:27:15] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556572.000000 [05:27:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 610879.000000 [05:27:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:28:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:14] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:14] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:14] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:28:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 688742.000000 [05:28:14] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:15] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:44] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:44] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:44] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:28:45] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:45] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:45] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:45] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:28:54] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:54] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:28:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:29:04] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:29:24] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:29:24] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:29:24] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:29:24] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64843 MB (10% inode=99%): [05:29:24] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:29:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:29:25] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:29:34] / on thyme is UNKNOWN: NRPE: Unable to read output [05:29:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17404 [05:29:34] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:29:34] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [05:29:34] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:29:35] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332433 MB (6% inode=63%): [05:29:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:29:36] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:29:44] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:29:44] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:29:44] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:29:44] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:30:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 277435.000000 [05:56:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:26:45] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:26:45] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:26:45] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:26:45] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [06:26:45] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:45] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:55] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:26:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 612247.000000 [06:26:55] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:55] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:26:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:55] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:26:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:26:56] aliasd on yarrow is CRITICAL: Connection refused [06:27:34] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:27:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:27:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:28:04] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:28:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18836.000000 [06:28:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:28:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:04] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:14] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556586.000000 [06:28:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 613173.000000 [06:28:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 590313.000000 [06:28:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:15] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:15] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:15] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:28:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 691986.000000 [06:28:16] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:17] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:28:45] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:45] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:28:45] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:45] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:54] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:54] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:54] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:54] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:55] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:28:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:29:04] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:24] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:29:24] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:29:24] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:29:24] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64710 MB (10% inode=99%): [06:29:24] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:29:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:29:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:29:34] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:29:34] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [06:29:34] / on thyme is UNKNOWN: NRPE: Unable to read output [06:29:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18932 [06:29:35] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:29:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332312 MB (6% inode=63%): [06:29:36] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:29:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:29:44] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:29:44] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:29:44] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:29:44] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:30:03] Load avg. on daphne is WARNING: WARNING - load average: 15.43, 15.15, 13.81 [06:30:14] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2162.000000 [06:30:25] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 267065.000000 [06:46:05] Load avg. on daphne is OK: OK - load average: 13.35, 14.79, 14.92 [06:46:15] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1751.000000 [06:54:53] aliasd on yarrow is OK: TCP OK - 0.013 second response time on port 984 [500 Not found.] [06:55:35] / on nightshade is OK: DISK OK - free space: / 1593 MB (89% inode=94%): [06:55:35] Sensors on nightshade is OK: sensor ok [06:55:36] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [06:55:45] /tmp on nightshade is OK: DISK OK - free space: /tmp 4310 MB (96% inode=99%): [06:55:45] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [06:55:45] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:55:54] Environment IPMI on nightshade is OK: ok: temperature ok fan ok voltage ok chassis ok [06:56:04] Load avg. on nightshade is OK: OK - load average: 1.69, 2.11, 2.13 [06:56:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:57:15] RAID on thyme is CRITICAL: Connection refused by host [06:58:15] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:06:45] aliasd on nightshade is CRITICAL: Connection refused [07:10:45] Free Memory on damiana is WARNING: WARNING - 6.9% (577728 kB) free! [07:18:44] aliasd on nightshade is OK: TCP OK - 0.004 second response time on port 984 [500 Not found.] [07:26:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:27:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:27:45] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:27:45] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [07:27:45] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (381168 kB) free! [07:27:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:55] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:27:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 613808.000000 [07:27:55] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:55] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:27:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:28:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20047.000000 [07:28:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:28:05] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:05] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:28:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:55] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:55] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:56] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:56] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:56] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:28:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:29:05] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 590571.000000 [07:29:15] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 554500.000000 [07:29:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 614884.000000 [07:29:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:15] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:16] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 694850.000000 [07:29:17] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:17] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:25] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:29:25] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:25] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:25] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64539 MB (10% inode=99%): [07:29:25] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:29:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:29:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:29:35] / on thyme is UNKNOWN: NRPE: Unable to read output [07:29:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20030 [07:29:35] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:29:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332103 MB (6% inode=63%): [07:29:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:29:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:29:45] Free Memory on damiana is WARNING: WARNING - 5.2% (438348 kB) free! [07:29:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:29:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:30:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 258403.000000 [07:33:45] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421768 kB) free! [07:41:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81489 MB (13% inode=99%): [07:54:44] aliasd on nightshade is CRITICAL: Connection refused [07:55:45] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (382480 kB) free! [07:56:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:56:44] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [07:58:15] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:03:44] aliasd on nightshade is CRITICAL: Connection refused [08:26:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:27:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:27:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:28:44] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:28:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:44] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [08:28:54] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:28:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:28:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 614078.000000 [08:28:54] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:28:54] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:28:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:29:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21062.000000 [08:29:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:29:04] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:05] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:05] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:29:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:29:35] / on thyme is UNKNOWN: NRPE: Unable to read output [08:29:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21064 [08:29:35] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:29:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332030 MB (6% inode=63%): [08:29:36] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:29:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:29:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:54] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:54] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:54] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:54] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:55] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:55] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:29:56] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:30:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 592174.000000 [08:30:14] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 551222.000000 [08:30:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 615837.000000 [08:30:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:14] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:15] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 697693.000000 [08:30:16] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:16] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:24] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:30:24] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:24] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 250831.000000 [08:30:24] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:30:53] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:54] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:30:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:34:44] Free Memory on damiana is WARNING: WARNING - 5.1% (428980 kB) free! [08:40:44] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422120 kB) free! [08:41:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81329 MB (13% inode=99%): [08:51:01] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:01] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 592755.000000 [08:51:01] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 615814.000000 [08:51:01] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 549087.000000 [08:51:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 698510.000000 [08:51:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:06] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 247309.000000 [08:51:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81312 MB (13% inode=99%): [08:51:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:51:07] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:07] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:51:25] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:58:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:16:25] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417952 kB) free! [09:21:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:50:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:50:16] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:50:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:50:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331701 MB (6% inode=63%): [09:50:17] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:50:26] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:50:26] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [09:50:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:26] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:50:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:36] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 614854.000000 [09:50:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:37] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:50:37] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:38] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:50:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:50:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23124.000000 [09:50:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:46] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:47] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:47] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:51:06] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 238904.000000 [09:51:06] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:06] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:16] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:51:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 542235.000000 [09:51:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616635.000000 [09:51:25] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 593403.000000 [09:51:56] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:51:56] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:51:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 701508.000000 [09:51:56] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:51:57] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:52:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81158 MB (13% inode=99%): [09:52:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:52:07] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:52:07] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:52:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:52:07] / on thyme is UNKNOWN: NRPE: Unable to read output [09:52:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23202 [09:56:06] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 28393 MB (5% inode=99%): [09:58:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:17:25] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (397160 kB) free! [10:21:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:25:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 21954 MB (2% inode=99%): [10:29:26] Free Memory on damiana is WARNING: WARNING - 5.1% (423248 kB) free! [10:30:25] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (414420 kB) free! [10:34:26] Free Memory on damiana is WARNING: WARNING - 5.4% (450192 kB) free! [10:50:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:50:15] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:50:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:50:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331605 MB (6% inode=63%): [10:50:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:50:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:50:26] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [10:50:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:26] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:50:37] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:37] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:37] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 614919.000000 [10:50:37] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:37] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:50:37] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:37] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:50:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:50:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26472.000000 [10:50:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:46] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:47] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:56] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:56] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:56] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:56] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:56] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:57] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:58] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:51:06] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 230199.000000 [10:51:06] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:51:06] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:51:15] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:51:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 535921.000000 [10:51:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616657.000000 [10:51:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:51:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 593685.000000 [10:51:56] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:51:56] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:51:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 704175.000000 [10:51:56] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:51:57] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:52:03] * Moe_Epsilon waves and leaves [10:52:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81012 MB (13% inode=99%): [10:52:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:52:07] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:52:07] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:52:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:52:07] / on thyme is UNKNOWN: NRPE: Unable to read output [10:52:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26559 [10:58:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:03:25] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421412 kB) free! [11:50:35] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:50:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:50:36] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:50:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 615865.000000 [11:51:06] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 221750.000000 [11:51:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:51:06] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:51:06] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:51:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:51:15] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:51:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:51:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331465 MB (6% inode=63%): [11:51:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:51:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:51:25] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [11:51:25] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:25] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:35] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:35] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:51:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:51:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:51:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27041.000000 [11:51:45] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:45] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:46] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:56] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:57] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 706006.000000 [11:51:58] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:58] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:52:15] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:52:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 532265.000000 [11:52:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616686.000000 [11:52:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:52:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 593058.000000 [11:52:56] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:52:56] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:52:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80818 MB (13% inode=99%): [11:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:53:05] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:53:05] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:53:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:53:06] / on thyme is UNKNOWN: NRPE: Unable to read output [11:53:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27127 [11:59:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:03:26] Free Memory on damiana is CRITICAL: CRITICAL - 2.8% (235712 kB) free! [12:24:50] @replag [12:24:51] Betacommand: s1-rr-a-wd: 5d 2h 54m 47s [-0.16 s/s]; s1-user: 7h 27m 26s [+0.14 s/s]; s1-user-wd: 1w 1d 4h 34m 20s [+0.61 s/s]; s2-user-c: error; s2-user-wd: 6d 21h 4m 15s [+0.38 s/s]; s3-user: 1m 2s [-0.00 s/s]; s3-user-wd: 6d 20m 13s [+0.16 s/s]; s4-user-wd: 1w 3h 18m 37s [+0.51 s/s] [12:24:52] Betacommand: s5-rr-a-wd: 2d 12h 8m 27s [-0.83 s/s]; s5-user: 2m 14s [+0.00 s/s]; s5-user-c: 37s [-0.11 s/s]; s6-user-wd: 1w 3h 19m 46s [+0.67 s/s]; s7-user-wd: 6d 3h 27m 31s [+0.05 s/s] [12:32:53] OK [12:32:59] So why are a lot ofthings breaking? [12:35:30] Qcoder00: maintenance that went bad [12:35:39] OK [12:36:30] Qcoder00: all of the servers needed new ip addresses [12:50:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:50:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:36] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 617622.000000 [12:51:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:51:07] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 213170.000000 [12:51:07] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:51:07] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:51:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:51:16] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:51:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:51:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331383 MB (6% inode=63%): [12:51:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:51:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:51:25] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [12:51:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:26] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:35] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:35] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:51:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:51:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26308.000000 [12:51:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:51:46] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:47] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:47] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:56] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:56] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:56] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:56] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:58] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:58] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:59] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 708707.000000 [12:51:59] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:52:00] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:52:15] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:52:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 529683.000000 [12:52:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616841.000000 [12:52:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:52:56] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:52:56] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:52:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 594840.000000 [12:53:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80599 MB (13% inode=99%): [12:53:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:53:06] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:53:06] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:53:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:53:07] / on thyme is UNKNOWN: NRPE: Unable to read output [12:53:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26321 [12:59:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:03:26] Free Memory on damiana is CRITICAL: CRITICAL - 3.0% (252772 kB) free! [13:22:04] John * [Toolserver-l] HTTP outage [13:31:04] Marc A. Pelletier * Re: [Toolserver-l] HTTP outage [13:40:04] DeltaQuad Wikipedia * Re: [Toolserver-l] HTTP outage [13:50:35] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:50:36] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 618874.000000 [13:51:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:51:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:51:16] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:51:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:51:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331300 MB (6% inode=63%): [13:51:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:51:35] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:35] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:51:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:51:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24218.000000 [13:51:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:05] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 205336.000000 [13:52:05] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:52:05] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:52:15] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:52:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:52:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 617102.000000 [13:52:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 525873.000000 [13:52:26] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [13:52:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:52:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:27] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:52:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:46] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:46] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 596830.000000 [13:52:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 709149.000000 [13:52:56] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:57] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:53:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80383 MB (13% inode=99%): [13:53:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:53:06] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:53:06] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:53:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:53:06] / on thyme is UNKNOWN: NRPE: Unable to read output [13:53:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24215 [13:53:55] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:53:55] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:53:55] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:03:26] Free Memory on damiana is CRITICAL: CRITICAL - 3.4% (287572 kB) free! [14:50:35] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:50:35] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 619644.000000 [14:51:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:51:17] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:51:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:51:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:51:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331121 MB (6% inode=63%): [14:51:17] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:51:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:36] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:51:36] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:51:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21767.000000 [14:51:56] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:56] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:56] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:05] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 198049.000000 [14:52:06] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:52:06] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:52:16] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:52:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:52:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 617144.000000 [14:52:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 521820.000000 [14:52:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:52:26] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [14:52:27] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:27] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:52:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:46] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:46] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:56] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:56] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:57] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:57] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 598661.000000 [14:52:57] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 710435.000000 [14:52:57] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:57] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:53:56] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:53:56] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:53:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80184 MB (13% inode=99%): [14:54:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:54:06] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:54:06] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:54:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:54:07] / on thyme is UNKNOWN: NRPE: Unable to read output [14:54:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21897 [15:00:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:04:25] Free Memory on damiana is CRITICAL: CRITICAL - 3.0% (250800 kB) free! [15:51:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:51:34] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:51:35] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:51:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:51:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 620871.000000 [15:52:06] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 188860.000000 [15:52:06] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:52:06] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:52:15] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:52:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:52:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:52:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330992 MB (6% inode=63%): [15:52:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:52:35] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:35] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:52:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:52:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19116.000000 [15:52:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:52:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:47] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:47] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:15] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:53:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:53:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518669.000000 [15:53:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616484.000000 [15:53:25] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:26] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [15:53:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:53:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:26] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 599932.000000 [15:53:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 711839.000000 [15:53:57] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:57] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:54:54] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:54:55] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:54:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80015 MB (13% inode=99%): [15:55:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:55:06] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:55:06] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:55:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:55:06] / on thyme is UNKNOWN: NRPE: Unable to read output [15:55:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18846 [16:01:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:04:26] Free Memory on damiana is CRITICAL: CRITICAL - 2.6% (216472 kB) free! [16:51:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:51:35] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:51:35] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:51:35] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:51:35] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 622145.000000 [16:52:05] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 179664.000000 [16:52:05] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:52:06] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:52:15] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:52:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:52:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:52:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330868 MB (6% inode=63%): [16:52:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:52:35] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:35] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:52:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:15] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:53:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:53:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 516255.000000 [16:53:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616093.000000 [16:53:25] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [16:53:25] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:53:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15687.000000 [16:53:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:53:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:46] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:46] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:56] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:55] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 600834.000000 [16:54:55] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:55] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 712639.000000 [16:54:56] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:57] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:55:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79791 MB (13% inode=99%): [16:55:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:55:06] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:55:06] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:55:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:55:07] / on thyme is UNKNOWN: NRPE: Unable to read output [16:55:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15667 [17:02:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:05:25] Free Memory on damiana is CRITICAL: CRITICAL - 3.3% (272800 kB) free! [17:51:34] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:51:35] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:51:35] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:51:35] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 622011.000000 [17:52:06] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 176500.000000 [17:52:06] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:52:06] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:52:35] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:35] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:52:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:14] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:53:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:53:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:53:16] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:53:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330566 MB (6% inode=63%): [17:53:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:53:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:53:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 614938.000000 [17:53:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 515191.000000 [17:53:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:53:26] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [17:53:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13448.000000 [17:53:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:53:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:53:45] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:45] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:45] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:55] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:17] Coren: is there a list of people with labs accounts? [17:54:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:45] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:55] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:55] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 601524.000000 [17:54:55] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:55] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713871.000000 [17:54:56] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:57] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79542 MB (13% inode=99%): [17:56:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:56:05] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:06] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:56:06] / on thyme is UNKNOWN: NRPE: Unable to read output [17:56:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13297 [17:56:44] anyone know if the toolserver is going to switch to MariaDB too? [18:00:25] Betacommand: https://wikitech.wikimedia.org/w/index.php?title=Special%3AListUsers&username=&group=shell&limit=50 [18:02:04] Tim Landscheidt * Re: [Toolserver-l] Failed Maintenance - was - Re: IP Renumbering of the complete Toolserver cluster (fwd) [18:02:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:03:18] Coren: +shell means you have labs access? [18:03:49] * Coren nods. [18:05:25] Free Memory on damiana is CRITICAL: CRITICAL - 3.2% (265024 kB) free! [18:07:42] gotta love the differences between the same term in different areas [18:10:42] Betacommand: The fun things with standards is that there are so many to choose from! [18:21:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:38:34] @replag [18:38:36] Betacommand: s1-rr-a-wd: 5d 2h 23m 32s [-0.08 s/s]; s1-user: 3h 20m 20s [-0.66 s/s]; s1-user-c: 45s [-]; s1-user-wd: 1w 1d 6h 35m 10s [+0.32 s/s]; s2-user-c: error; s2-user-wd: 6d 23h 10m 8s [+0.34 s/s]; s3-user: 1m 23s [+0.00 s/s]; s3-user-wd: 5d 21h 13m 5s [-0.50 s/s] [18:38:37] Betacommand: s4-rr-a: 12s [-]; s4-user: 12s [-]; s4-user-wd: 1w 4h 53m 33s [+0.25 s/s]; s5-rr-a-wd: 2d 23m 27s [-1.89 s/s]; s6-user-wd: 1w 2h 48m 20s [-0.08 s/s]; s7-user-wd: 5d 23h 10m 12s [-0.69 s/s] [18:45:30] [[Special:Log/newusers]] create 10 * Tjpolchow19 * (New user account) [18:51:35] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:51:36] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 622582.000000 [18:52:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:36] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:52:36] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:56] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:56] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:56] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:07] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 173655.000000 [18:53:07] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:53:07] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:53:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:53:16] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:53:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:53:16] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:53:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330360 MB (6% inode=63%): [18:53:17] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:53:26] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:53:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 515383.000000 [18:53:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 615075.000000 [18:53:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:53:27] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [18:53:27] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11707.000000 [18:53:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:53:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:53:46] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:47] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:56] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:57] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:46] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 601765.000000 [18:54:56] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:56] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:57] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715186.000000 [18:54:57] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:58] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79320 MB (13% inode=99%): [18:56:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:56:06] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:06] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:56:07] / on thyme is UNKNOWN: NRPE: Unable to read output [18:56:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11685 [19:02:57] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:05:26] Free Memory on damiana is CRITICAL: CRITICAL - 2.7% (229880 kB) free! [19:21:04] DaB. * [Toolserver-l] Flue again [19:51:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:51:35] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:51:35] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:35] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:35] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 623856.000000 [19:52:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:52:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:52:56] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:53:05] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:53:05] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 170668.000000 [19:53:05] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:53:05] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:53:35] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:36] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:53:36] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:53:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:53:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:53:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10569.000000 [19:53:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:53:46] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:53:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:53:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:16] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:54:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:54:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:54:16] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:54:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330104 MB (6% inode=63%): [19:54:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:54:25] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:54:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 514300.000000 [19:54:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616161.000000 [19:54:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:54:26] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [19:54:27] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:27] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:46] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:57] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 600917.000000 [19:54:57] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:57] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:58] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 717093.000000 [19:54:58] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:59] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:55:00] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:55:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:55:05] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79127 MB (12% inode=99%): [19:56:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:56:07] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:07] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:56:08] / on thyme is UNKNOWN: NRPE: Unable to read output [19:56:08] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10452 [20:02:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:05:26] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (238936 kB) free! [20:22:05] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 169121.000000 [20:22:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78985 MB (12% inode=99%): [20:22:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:22:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:22:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9238 [20:22:05] / on thyme is UNKNOWN: NRPE: Unable to read output [20:22:14] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:27:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 600405.000000 [20:27:30] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616796.000000 [20:27:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 514368.000000 [20:27:30] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:27:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 718104.000000 [20:27:35] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 168839.000000 [20:27:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78970 MB (12% inode=99%): [20:27:35] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:27:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:27:36] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:27:36] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:27:36] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:27:36] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9283 [20:27:37] / on thyme is UNKNOWN: NRPE: Unable to read output [20:27:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:27:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:27:47] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:27:47] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:27:47] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:27:48] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:27:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:27:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:33:36] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 600242.000000 [20:33:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616931.000000 [20:33:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 514327.000000 [20:33:36] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:33:36] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:33:37] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:37] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:33:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 718400.000000 [20:33:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:33:39] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:33:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:33:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 168589.000000 [20:33:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78960 MB (12% inode=99%): [20:33:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:32:39] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:32:39] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:32:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:32:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:32:59] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [21:32:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:32:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:32:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:32:59] Free Memory on damiana is CRITICAL: CRITICAL - 2.3% (193760 kB) free! [21:33:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:33:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78684 MB (12% inode=99%): [21:33:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 164917.000000 [21:33:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:33:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:33:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:33:40] / on thyme is UNKNOWN: NRPE: Unable to read output [21:33:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7887 [21:33:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:33:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:33:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:33:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:33:49] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:33:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:33:50] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:33:51] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 329953 MB (6% inode=63%): [21:33:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:33:52] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:33:52] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:33:53] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:33:53] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:33:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:34:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 625875.000000 [21:34:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:10] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:34:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:34:11] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:34:19] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7848.000000 [21:34:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:19] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:20] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 599541.000000 [21:34:33] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 513245.000000 [21:34:33] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 618456.000000 [21:34:33] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:33] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:33] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:34:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 720642.000000 [21:34:33] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:34:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:56:39] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 27149 MB (5% inode=99%): [22:25:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 21957 MB (2% inode=99%): [22:32:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:32:40] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:32:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:32:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:32:59] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [22:32:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:32:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:32:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:32:59] Free Memory on damiana is CRITICAL: CRITICAL - 2.3% (196272 kB) free! [22:33:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:33:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:33:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78433 MB (12% inode=99%): [22:33:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 161086.000000 [22:33:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:33:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:33:40] / on thyme is UNKNOWN: NRPE: Unable to read output [22:33:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6188 [22:33:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:33:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:33:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:33:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:33:49] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:33:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:33:50] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:33:51] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 329932 MB (6% inode=63%): [22:33:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:33:52] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:33:52] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:33:53] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:33:53] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:33:54] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:34:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:34:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:34:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:34:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 628123.000000 [22:34:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:34:19] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6169.000000 [22:34:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:34:19] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:34:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:34:20] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 597233.000000 [22:34:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 509862.000000 [22:34:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 619995.000000 [22:34:29] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:34:30] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:35:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:35:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:35:09] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:35:09] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:35:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:29] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:35:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 722122.000000 [22:35:29] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:35:29] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:40] how can I tell if my toolserver accoutn has been approved? [23:07:46] greenrosetta: you would get an email [23:09:47] wtf [23:09:53] toolserver constantly logs me out [23:25:19] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3593.000000 [23:25:39] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3561 [23:28:20] FC 0/22 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [23:29:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:32:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:32:41] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:32:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:33:00] APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates). [23:33:00] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:00] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:01] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:01] Free Memory on damiana is CRITICAL: CRITICAL - 2.2% (184120 kB) free! [23:33:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157032.000000 [23:33:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78199 MB (12% inode=99%): [23:33:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:33:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:33:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:33:40] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:33:41] / on thyme is UNKNOWN: NRPE: Unable to read output [23:33:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:33:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:33:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:49] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:33:49] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 328975 MB (6% inode=63%): [23:33:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:33:50] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:51] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:51] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:52] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:33:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:53] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:33:53] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:34:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 630534.000000 [23:34:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:34:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:34:19] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:19] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:34:28] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:29] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506112.000000 [23:34:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 621575.000000 [23:34:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 592908.000000 [23:34:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:34:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:34:49] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:34:50] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:35:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:35:09] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:35:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:35:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 723878.000000 [23:35:29] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:35:29] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:35:29] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:40:39] MySQL slave on rosemary is OK: Uptime: 1671382 Threads: 10 Questions: 1432008354 Slow queries: 506208 Opens: 74473 Flush tables: 1 Open tables: 3515 Queries per second avg: 856.781 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1794 [23:41:19] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1740.000000