[00:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [00:06:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [00:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [00:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [00:07:15] SMTP on turnera is CRITICAL: Connection refused [00:07:15] Free Memory on turnera is CRITICAL: Connection refused by host [00:07:15] / on turnera is CRITICAL: Connection refused by host [00:07:15] DiskSuite on turnera is CRITICAL: Connection refused by host [00:07:15] ts-array5 on turnera is CRITICAL: Connection refused by host [00:07:15] /tmp on turnera is CRITICAL: Connection refused by host [00:53:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:53:08] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1590796.000000 [00:53:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1747598.000000 [00:53:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:53:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:53:56] nacht ts [00:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [01:06:58] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [01:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [01:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [01:07:15] SMTP on turnera is CRITICAL: Connection refused [01:07:15] Free Memory on turnera is CRITICAL: Connection refused by host [01:07:15] DiskSuite on turnera is CRITICAL: Connection refused by host [01:07:15] / on turnera is CRITICAL: Connection refused by host [01:07:15] /tmp on turnera is CRITICAL: Connection refused by host [01:07:15] ts-array5 on turnera is CRITICAL: Connection refused by host [01:16:29] TOOLSERVER's BROKEN AGAIN! :p [01:20:43] @replag [01:24:23] Cyberpower678: What's not working? Login, SGE and web work for me. [01:26:01] scfc_de, Some cluster and it's freakin' slow. [01:26:38] Cyberpower678: The only thing I see is that SGE isn't running any jobs. [01:27:19] I can't load the SUL, global contribs, and the edit counter tool and ACC is barely working. [01:28:32] URLs? [01:29:18] I can't. ACC stopped working now. [01:29:48] No. I can't link to it because it hosts private data. [01:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:31:52] Well, the edit counter is probably not private, is it? [01:35:22] task doesn't start, they are in qw state since hours [01:39:59] tons of defunct task on nightshade, weird user name like '1187', many /USR/SBIN/CRON owned by root [01:42:51] 1187 seems to be DaBPunkt and his query killer. [01:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:27] The multiple /USR/SBIN/CRON look really odd. [01:45:27] 61 /USR/SBIN/CRON task, 45 task owned by 1187 [01:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:58] Sorry, query-killer is by dispenser. [01:46:04] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:46:38] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:38] 61 task defunct [01:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:02] there is other things owned by 1187 which are not the query killer [01:47:15] SSH on mayapple is CRITICAL: Server answer: [01:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:46] Bryan's stuff seems to be WSCGI, some daemon for CGIs. [01:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 210430.000000 [01:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:49:14] Hmmm. If I "stat /home/dispenser", 1187 is correctly translated to dispenser. Why not on ps? [01:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:24] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741972.000000 [01:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [01:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [01:50:05] bryan has 56 time the same script started [01:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210477 [01:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:50:26] Yeah, but I don't know if this is a bug on his part or normal procedure for WSCGIs. [01:50:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 938820.000000 [01:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241969 MB (4% inode=63%): [01:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:51:00] Anyway, I don't think that this is related to SGE not scheduling new jobs. [01:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [01:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2853827.000000 [01:51:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1902244.000000 [01:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [01:52:05] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [01:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:53:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:53:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1589391.000000 [01:53:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:53:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1751199.000000 [01:55:04] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:55:59] Well: Good night. [02:06:19] Load avg. on turnera is CRITICAL: Connection refused by host [02:06:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [02:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [02:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [02:07:15] SMTP on turnera is CRITICAL: Connection refused [02:07:15] DiskSuite on turnera is CRITICAL: Connection refused by host [02:07:16] /tmp on turnera is CRITICAL: Connection refused by host [02:07:16] Free Memory on turnera is CRITICAL: Connection refused by host [02:07:16] ts-array5 on turnera is CRITICAL: Connection refused by host [02:07:16] / on turnera is CRITICAL: Connection refused by host [02:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:44:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:46:34] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:15] SSH on mayapple is CRITICAL: Server answer: [02:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:48:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 211359.000000 [02:48:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [02:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741706.000000 [02:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [02:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [02:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 211381 [02:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:50:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 937523.000000 [02:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241933 MB (4% inode=63%): [02:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [02:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2857427.000000 [02:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1905781.000000 [02:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [02:52:06] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [02:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1588179.000000 [02:53:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:53:06] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1754800.000000 [02:55:04] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:58:24] Cluster S1 is dead [03:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [03:06:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [03:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [03:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [03:07:15] SMTP on turnera is CRITICAL: Connection refused [03:07:16] DiskSuite on turnera is CRITICAL: Connection refused by host [03:07:16] /tmp on turnera is CRITICAL: Connection refused by host [03:07:16] / on turnera is CRITICAL: Connection refused by host [03:07:16] Free Memory on turnera is CRITICAL: Connection refused by host [03:07:16] ts-array5 on turnera is CRITICAL: Connection refused by host [03:27:55] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [03:27:55] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [03:27:55] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [03:27:55] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [03:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:46:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:15] SSH on mayapple is CRITICAL: Server answer: [03:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 211477.000000 [03:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:49:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 740780.000000 [03:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [03:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [03:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 211481 [03:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:50:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241882 MB (4% inode=63%): [03:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 936415.000000 [03:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:51:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [03:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2861027.000000 [03:51:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1909326.000000 [03:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [03:52:05] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [03:52:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:53:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [03:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1758400.000000 [03:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [04:06:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [04:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [04:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [04:07:15] SMTP on turnera is CRITICAL: Connection refused [04:07:15] ts-array5 on turnera is CRITICAL: Connection refused by host [04:07:15] Free Memory on turnera is CRITICAL: Connection refused by host [04:07:15] /tmp on turnera is CRITICAL: Connection refused by host [04:07:15] DiskSuite on turnera is CRITICAL: Connection refused by host [04:07:15] / on turnera is CRITICAL: Connection refused by host [04:27:54] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [04:27:55] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [04:27:55] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [04:27:56] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [04:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:45] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:46:34] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:44] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:16] SSH on mayapple is CRITICAL: Server answer: [04:47:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:45] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 211905.000000 [04:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:49:04] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:49:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739980.000000 [04:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [04:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [04:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 211903 [04:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:50:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241832 MB (4% inode=63%): [04:50:56] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 934987.000000 [04:50:56] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [04:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2864627.000000 [04:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1912850.000000 [04:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [04:52:05] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [04:52:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [04:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1762000.000000 [04:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [05:06:54] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [05:07:04] Environment IPMI on turnera is CRITICAL: Connection refused by host [05:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [05:07:15] SMTP on turnera is CRITICAL: Connection refused [05:07:15] Free Memory on turnera is CRITICAL: Connection refused by host [05:07:15] ts-array5 on turnera is CRITICAL: Connection refused by host [05:07:16] / on turnera is CRITICAL: Connection refused by host [05:07:16] /tmp on turnera is CRITICAL: Connection refused by host [05:07:16] DiskSuite on turnera is CRITICAL: Connection refused by host [05:27:55] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [05:27:55] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [05:27:55] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [05:27:55] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [05:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:46:04] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:46:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:15] SSH on mayapple is CRITICAL: Server answer: [05:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 211783.000000 [05:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [05:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739927.000000 [05:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [05:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [05:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 211780 [05:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:50:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241803 MB (4% inode=63%): [05:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 933698.000000 [05:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:51:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [05:51:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2868227.000000 [05:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1916409.000000 [05:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [05:52:05] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [05:52:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:53:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [05:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1765600.000000 [05:54:10] @replag [05:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [06:06:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [06:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [06:07:06] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [06:07:14] SMTP on turnera is CRITICAL: Connection refused [06:07:15] /tmp on turnera is CRITICAL: Connection refused by host [06:07:15] ts-array5 on turnera is CRITICAL: Connection refused by host [06:07:15] Free Memory on turnera is CRITICAL: Connection refused by host [06:07:16] / on turnera is CRITICAL: Connection refused by host [06:08:15] DiskSuite on turnera is CRITICAL: Connection refused by host [06:27:55] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [06:27:55] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [06:27:55] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [06:27:55] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [06:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:46:34] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:15] SSH on mayapple is CRITICAL: Server answer: [06:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 212383.000000 [06:48:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:48:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:48:55] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [06:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:49:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:44] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739437.000000 [06:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [06:49:56] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [06:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 212368 [06:50:14] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:50:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 933663.000000 [06:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241615 MB (4% inode=63%): [06:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [06:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2871827.000000 [06:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1919964.000000 [06:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [06:52:05] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [06:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [06:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1769200.000000 [06:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [07:06:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [07:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [07:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [07:07:15] SMTP on turnera is CRITICAL: Connection refused [07:07:15] /tmp on turnera is CRITICAL: Connection refused by host [07:07:15] Free Memory on turnera is CRITICAL: Connection refused by host [07:07:15] ts-array5 on turnera is CRITICAL: Connection refused by host [07:07:15] / on turnera is CRITICAL: Connection refused by host [07:08:14] DiskSuite on turnera is CRITICAL: Connection refused by host [07:24:05] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [07:24:15] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [07:27:54] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [07:27:55] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [07:27:55] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [07:27:56] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [07:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:40:05] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 197.00 ms [07:44:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:45:45] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:46:05] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 141.00 ms [07:46:34] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:48] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:48] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:15] SSH on mayapple is CRITICAL: Server answer: [07:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:45] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 212684.000000 [07:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:48:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:24] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:48:55] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [07:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:49:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739686.000000 [07:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [07:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [07:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 212619 [07:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:50:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241533 MB (4% inode=63%): [07:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 932723.000000 [07:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [07:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2875427.000000 [07:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1923522.000000 [07:51:54] / on thyme is UNKNOWN: NRPE: Unable to read output [07:52:04] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [07:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [07:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1772800.000000 [07:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [08:06:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [08:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [08:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [08:07:15] SMTP on turnera is CRITICAL: Connection refused [08:07:15] / on turnera is CRITICAL: Connection refused by host [08:07:15] Free Memory on turnera is CRITICAL: Connection refused by host [08:07:15] /tmp on turnera is CRITICAL: Connection refused by host [08:07:15] ts-array5 on turnera is CRITICAL: Connection refused by host [08:08:15] DiskSuite on turnera is CRITICAL: Connection refused by host [08:24:05] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [08:24:15] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [08:27:55] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [08:27:55] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [08:27:56] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [08:27:56] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [08:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:35:25] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [08:36:05] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [08:36:15] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [08:44:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:00] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:46:45] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:55] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:55] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:15] SSH on mayapple is CRITICAL: Server answer: [08:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:54] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:48:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 212899.000000 [08:48:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:48:55] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [08:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:55] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:56] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:56] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:56] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:56] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:49:56] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [08:49:56] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 740317.000000 [08:49:57] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [08:50:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 212920 [08:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:50:55] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931751.000000 [08:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241399 MB (4% inode=63%): [08:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [08:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2879027.000000 [08:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1927082.000000 [08:51:54] / on thyme is UNKNOWN: NRPE: Unable to read output [08:52:05] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [08:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [08:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1776400.000000 [08:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:13:14] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [09:23:27] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 174.00 ms [09:24:15] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 141.00 ms [09:27:55] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [09:27:55] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [09:27:55] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [09:27:55] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [09:29:14] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [09:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:42:15] PING on amaranth is WARNING: PING WARNING - Packet loss = 0%, RTA = 169.00 ms [09:44:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:55] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:54] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:55] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:15] SSH on mayapple is CRITICAL: Server answer: [09:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:55] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 212715.000000 [09:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:48:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:24] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:55] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [09:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:55] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:55] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:55] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:55] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:50:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 212704 [09:50:17] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:50:55] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [09:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:50:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 740589.000000 [09:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 241191 MB (4% inode=63%): [09:50:56] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 930739.000000 [09:50:56] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [09:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [09:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2882627.000000 [09:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1930633.000000 [09:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [09:52:04] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [09:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [09:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1780000.000000 [09:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:53:15] PING on amaranth is OK: PING OK - Packet loss = 0%, RTA = 141.00 ms [09:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:58:34] [[Special:Log/newusers]] create 10 * Sdule1 * (New user account) [10:27:54] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [10:27:55] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [10:27:56] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [10:27:56] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [10:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:44:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:45:54] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:55] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:55] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:15] SSH on mayapple is CRITICAL: Server answer: [10:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:47:55] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 212975.000000 [10:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:48:55] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [10:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:49:59] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:49:59] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:49:59] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:49:59] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:50:05] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 212997 [10:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:50:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [10:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:50:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239309 MB (4% inode=63%): [10:50:56] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741065.000000 [10:50:56] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 930772.000000 [10:50:56] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [10:50:56] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:51:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [10:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2886226.000000 [10:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1932817.000000 [10:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [10:52:05] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [10:52:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [10:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1783600.000000 [10:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:53:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:55:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:27:55] MySQL on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [11:27:55] MySQL on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [11:27:55] MySQL slave on z-dat-s1-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [11:27:55] MySQL slave on z-dat-s2-b is CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [11:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:45:05] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:05] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:04] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:05] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:15] SSH on mayapple is CRITICAL: Server answer: [11:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:48:05] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 213743.000000 [11:48:05] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (355700 kB) free! [11:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:48:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:24] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:24] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:54] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:48:55] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [11:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:49:55] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:06] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:06] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:06] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:06] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 213771 [11:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:50:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [11:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:50:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 740934.000000 [11:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 932296.000000 [11:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 240594 MB (4% inode=63%): [11:50:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [11:51:05] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [11:51:06] Free Memory on damiana is OK: OK - 12.8% (1069332 kB) free. [11:51:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2889827.000000 [11:51:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1932315.000000 [11:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [11:52:05] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [11:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Host damiana-bge0.esi.toolserver.org is blocked because of many connection errors: unblock with mysqladmin flush-hosts [11:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1787200.000000 [11:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:56:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:01:20] Hello all [12:15:55] MySQL on z-dat-s2-b is OK: Uptime: 276230 Threads: 5 Questions: 189002201 Slow queries: 4614 Opens: 1303978 Flush tables: 1 Open tables: 256 Queries per second avg: 684.220 [12:15:56] MySQL slave on z-dat-s2-b is OK: Uptime: 276232 Threads: 4 Questions: 189003914 Slow queries: 4614 Opens: 1303982 Flush tables: 1 Open tables: 256 Queries per second avg: 684.221 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:17:55] MySQL slave on z-dat-s1-b is OK: Uptime: 301970 Threads: 7 Questions: 229397927 Slow queries: 6114 Opens: 1507 Flush tables: 1 Open tables: 542 Queries per second avg: 759.671 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:17:56] MySQL on z-dat-s1-b is OK: Uptime: 301970 Threads: 7 Questions: 229397912 Slow queries: 6114 Opens: 1507 Flush tables: 1 Open tables: 542 Queries per second avg: 759.671 [12:19:05] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (405024 kB) free! [12:24:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [12:24:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [12:24:15] SMTP on turnera is CRITICAL: Connection refused [12:24:34] / on turnera is CRITICAL: Connection refused by host [12:24:35] ts-array5 on turnera is CRITICAL: Connection refused by host [12:24:35] Free Memory on turnera is CRITICAL: Connection refused by host [12:24:35] /tmp on turnera is CRITICAL: Connection refused by host [12:24:35] Load avg. on turnera is CRITICAL: Connection refused by host [12:24:36] DiskSuite on turnera is CRITICAL: Connection refused by host [12:24:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [12:26:14] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 65321 MB (6% inode=99%): [12:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:43:05] Free Memory on damiana is WARNING: WARNING - 5.3% (440464 kB) free! [12:44:05] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (364176 kB) free! [12:45:05] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:05] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:45] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:05] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:05] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:15] SSH on mayapple is CRITICAL: Server answer: [12:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:48:04] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 214270.000000 [12:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [12:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:55] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [12:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:49:05] Free Memory on damiana is WARNING: WARNING - 5.8% (486960 kB) free! [12:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:49:24] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:49:55] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:50:08] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:50:08] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:50:08] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:50:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:50:08] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 214287 [12:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:50:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [12:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:50:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739701.000000 [12:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 933484.000000 [12:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 240332 MB (4% inode=63%): [12:50:56] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [12:51:05] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [12:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2893427.000000 [12:51:55] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [12:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [12:52:05] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [12:52:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1930878.000000 [12:52:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62624 MB (10% inode=99%): [12:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1565708.000000 [12:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1790799.000000 [12:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:53:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:56:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:00:46] johang: is there a way to fetch the list on https://toolserver.org/~johang/wikitrends/english-uptrends-today.html without scrapping? [13:10:55] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2098 [13:13:04] Free Memory on damiana is CRITICAL: CRITICAL - 3.7% (307440 kB) free! [13:17:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 595308.000000 [13:31:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:33:55] MySQL slave on z-dat-s2-b is OK: Uptime: 280912 Threads: 10 Questions: 191255538 Slow queries: 4763 Opens: 1318457 Flush tables: 1 Open tables: 256 Queries per second avg: 680.837 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1702 [13:43:16] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2808.000000 [13:45:05] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:05] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:44] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:05] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:06] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:47:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:16] SSH on mayapple is CRITICAL: Server answer: [13:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:48:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 212859.000000 [13:48:05] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:48:16] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:48:16] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [13:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:26] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:34] ts-array5 on turnera is CRITICAL: Connection refused by host [13:48:35] Load avg. on turnera is CRITICAL: Connection refused by host [13:48:36] DiskSuite on turnera is CRITICAL: Connection refused by host [13:48:36] Free Memory on turnera is CRITICAL: Connection refused by host [13:48:36] / on turnera is CRITICAL: Connection refused by host [13:48:36] /tmp on turnera is CRITICAL: Connection refused by host [13:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:48:55] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [13:48:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [13:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:49:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [13:49:06] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [13:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:49:16] SMTP on turnera is CRITICAL: Connection refused [13:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:50:05] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:50:05] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:50:05] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:50:05] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:50:05] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:50:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 212759 [13:50:19] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:50:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [13:50:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:50:57] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 240279 MB (4% inode=63%): [13:50:57] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 934572.000000 [13:50:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741037.000000 [13:50:58] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [13:51:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [13:51:06] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:51:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2897027.000000 [13:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [13:52:04] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [13:52:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1928365.000000 [13:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:53:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1794401.000000 [13:53:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1567184.000000 [13:53:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:56:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:41:39] re [15:54:37] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/4:UP:1 UP: OK [15:54:37] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [15:54:37] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:54:37] PING on asw-oe10-esams.mgmt is OK: PING OK - Packet loss = 0%, RTA = 1.27 ms [15:54:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 240279 MB (4% inode=63%): [15:54:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:54:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:54:47] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [15:54:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:54:47] PING on hemlock.mgmt is OK: PING OK - Packet loss = 0%, RTA = 0.61 ms [15:54:56] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:54:56] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:56] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:55:06] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:55:06] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:55:06] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:57:37] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:57:47] Free Memory on turnera is CRITICAL: Connection refused by host [15:57:47] Load avg. on turnera is CRITICAL: Connection refused by host [15:57:57] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:58:06] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [15:58:16] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:58:16] / on turnera is CRITICAL: Connection refused by host [15:58:17] ts-array5 on turnera is CRITICAL: Connection refused by host [15:58:26] /tmp on turnera is CRITICAL: Connection refused by host [15:58:26] DiskSuite on turnera is CRITICAL: Connection refused by host [15:58:36] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:58:56] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [15:59:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:59:07] SMTP on turnera is CRITICAL: Connection refused [15:59:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:59:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:59:07] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:59:27] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:59:37] Environment IPMI on turnera is CRITICAL: Connection refused by host [15:59:47] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5657 [15:59:47] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [15:59:47] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [15:59:47] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7078.000000 [15:59:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [15:59:57] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [15:59:57] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4806.000000 [16:00:16] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [16:00:17] aliasd on yarrow is CRITICAL: Connection refused [16:00:26] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 4.58, 20.85, 52.45 [16:00:26] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5317 [16:00:27] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4630 [16:00:27] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [16:00:36] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6161 [16:00:37] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2531 [16:00:37] Load avg. on yarrow is CRITICAL: CRITICAL - load average: 2.84, 18.75, 52.64 [16:01:17] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [16:01:56] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [16:02:16] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [16:02:37] MySQL slave on z-dat-s4-a is OK: Uptime: 1196710 Threads: 1 Questions: 57910152 Slow queries: 14 Opens: 93 Flush tables: 1 Open tables: 72 Queries per second avg: 48.391 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1124 [16:02:57] Sun Grid Engine execd on nightshade is CRITICAL: CRITICAL: execd not communicating [16:03:56] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [16:04:27] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3490 [16:04:56] war are back after a short outage [16:04:57] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3333.000000 [16:05:04] war → we [16:05:26] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3557 [16:07:47] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3424 [16:09:47] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3513.000000 [16:09:55] Why is toolserver collapsing like this? [16:10:26] MySQL slave on z-dat-s7-a is OK: Uptime: 1026177 Threads: 6 Questions: 835036095 Slow queries: 15819 Opens: 1081665 Flush tables: 1 Open tables: 3628 Queries per second avg: 813.734 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1722 [16:10:26] MySQL slave on daphne is OK: Uptime: 2492631 Threads: 18 Questions: 289963713 Slow queries: 528781 Opens: 86573 Flush tables: 1 Open tables: 1931 Queries per second avg: 116.328 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1759 [16:10:26] It's like an old man dying because his organs can no longer keep up. [16:10:56] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1629.000000 [16:12:52] MySQL slave on z-dat-s3-a is OK: Uptime: 1027138 Threads: 31 Questions: 1192210606 Slow queries: 30554 Opens: 6595217 Flush tables: 1 Open tables: 16384 Queries per second avg: 1160.711 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1800 [16:13:54] Cyberpower678: we should have reverted to Linux after river left. but didnt [16:14:16] And now it's too late? [16:14:35] Cyberpower678: No, in fact that is what will be happending [16:15:18] Cyberpower678: but since there have been solaris issues that have reared up and bitten us in the ass, its causing a disruption until its fixed [16:15:43] most of the current roots dont know solaris [16:15:47] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1623.000000 [16:16:36] Load avg. on yarrow is WARNING: WARNING - load average: 0.08, 0.87, 18.77 [16:17:27] Load avg. on nightshade is WARNING: WARNING - load average: 3.40, 3.46, 19.22 [16:20:19] But wasn't Nosy hired exactly because she had Solaris skills? [16:20:37] Load avg. on yarrow is OK: OK - load average: 0.39, 0.60, 14.57 [16:22:26] Load avg. on nightshade is OK: OK - load average: 3.42, 3.53, 14.89 [16:33:37] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3558 [16:41:36] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1525.000000 [16:43:21] Cyberpower678: see http://lists.wikimedia.org/pipermail/toolserver-l/2013-May/006057.html [16:43:36] MySQL slave on z-dat-s6-a is OK: Uptime: 1029166 Threads: 10 Questions: 845881808 Slow queries: 35642 Opens: 1366598 Flush tables: 1 Open tables: 3272 Queries per second avg: 821.909 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1793 [16:43:37] scfc_de: I dont think so [16:43:59] Betacommand, I already read it. [16:49:57] Betacommand: Seems you're right, neither http://permalink.gmane.org/gmane.org.wikimedia.toolserver/3730 nor http://permalink.gmane.org/gmane.org.wikimedia.toolserver.announce/1 mentions anything about Solaris. [16:53:57] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:53:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 183399.000000 [16:53:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [16:53:57] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:54:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2908007.000000 [16:54:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1934641.000000 [16:54:06] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [16:54:07] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:07] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:07] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:16] / on thyme is UNKNOWN: NRPE: Unable to read output [16:54:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:54:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1574907.000000 [16:54:17] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:17] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:17] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:18] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:54:26] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 183412 [16:54:27] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:27] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:54:27] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:37] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [16:54:37] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:54:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 240128 MB (4% inode=63%): [16:54:47] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:54:47] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:54:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:54:47] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [16:54:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 749173.000000 [16:54:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:47] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [16:54:48] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 943553.000000 [16:54:57] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:57] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1805317.000000 [16:54:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:55:07] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:55:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:07] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:55:07] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:55:27] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:55:27] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:27] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [16:55:27] SSH on mayapple is CRITICAL: Server answer: [16:55:27] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:55:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:57:46] Free Memory on turnera is CRITICAL: Connection refused by host [16:57:46] Load avg. on turnera is CRITICAL: Connection refused by host [16:58:17] ts-array5 on turnera is CRITICAL: Connection refused by host [16:58:17] / on turnera is CRITICAL: Connection refused by host [16:58:27] /tmp on turnera is CRITICAL: Connection refused by host [16:58:27] DiskSuite on turnera is CRITICAL: Connection refused by host [16:58:57] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [16:59:06] SMTP on turnera is CRITICAL: Connection refused [16:59:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:59:07] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:59:26] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:59:36] Environment IPMI on turnera is CRITICAL: Connection refused by host [16:59:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:59:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:59:47] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [16:59:47] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [16:59:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [17:00:17] aliasd on yarrow is CRITICAL: Connection refused [17:00:37] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [17:02:37] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [17:53:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 184389.000000 [17:53:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:53:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [17:53:58] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:54:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2911607.000000 [17:54:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1934614.000000 [17:54:07] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [17:54:07] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:07] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:54:07] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:54:17] / on thyme is UNKNOWN: NRPE: Unable to read output [17:54:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:54:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1575092.000000 [17:54:17] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:17] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:18] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:54:18] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:19] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:54:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 184391 [17:54:27] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:27] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:54:27] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:54:37] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [17:54:37] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:54:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239955 MB (4% inode=63%): [17:54:47] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:54:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:54:47] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [17:54:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:54:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 749901.000000 [17:54:48] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [17:54:48] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 944498.000000 [17:54:56] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:54:57] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1808917.000000 [17:54:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:06] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:07] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:07] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:26] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:26] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:27] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [17:55:27] SSH on mayapple is CRITICAL: Server answer: [17:55:28] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:55:36] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:57:46] Free Memory on turnera is CRITICAL: Connection refused by host [17:57:47] Load avg. on turnera is CRITICAL: Connection refused by host [17:58:17] / on turnera is CRITICAL: Connection refused by host [17:58:17] ts-array5 on turnera is CRITICAL: Connection refused by host [17:58:27] /tmp on turnera is CRITICAL: Connection refused by host [17:58:27] DiskSuite on turnera is CRITICAL: Connection refused by host [17:58:56] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [17:59:06] SMTP on turnera is CRITICAL: Connection refused [17:59:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:59:07] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:59:26] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:59:37] Environment IPMI on turnera is CRITICAL: Connection refused by host [17:59:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:59:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:59:47] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [17:59:47] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [17:59:47] User danmichaelo here? [17:59:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [18:00:17] aliasd on yarrow is CRITICAL: Connection refused [18:00:36] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [18:03:16] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [18:08:16] aliasd on yarrow is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [18:28:07] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (423092 kB) free! [18:29:07] Free Memory on damiana is WARNING: WARNING - 5.3% (447812 kB) free! [18:31:07] Free Memory on damiana is CRITICAL: CRITICAL - 3.9% (325452 kB) free! [18:53:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:53:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 184868.000000 [18:53:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [18:54:00] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:54:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2915208.000000 [18:54:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1934226.000000 [18:54:08] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:08] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [18:54:08] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:54:08] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:54:17] / on thyme is UNKNOWN: NRPE: Unable to read output [18:54:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:54:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1573874.000000 [18:54:17] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:18] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:54:18] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:18] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:18] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:54:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 184852 [18:54:27] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:27] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:54:27] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:54:37] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [18:54:38] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [18:54:38] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239759 MB (4% inode=63%): [18:54:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:54:47] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:54:47] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [18:54:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 750409.000000 [18:54:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:54:48] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [18:54:48] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 945582.000000 [18:54:57] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:54:57] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1812517.000000 [18:54:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:55:07] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:55:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:07] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:55:07] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:55:26] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:55:26] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:26] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:55:26] SSH on mayapple is CRITICAL: Server answer: [18:55:26] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [18:55:27] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:55:36] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:57:47] Free Memory on turnera is CRITICAL: Connection refused by host [18:57:47] Load avg. on turnera is CRITICAL: Connection refused by host [18:58:07] Free Memory on damiana is CRITICAL: CRITICAL - 3.8% (319644 kB) free! [18:58:17] ts-array5 on turnera is CRITICAL: Connection refused by host [18:58:17] / on turnera is CRITICAL: Connection refused by host [18:58:27] /tmp on turnera is CRITICAL: Connection refused by host [18:58:27] DiskSuite on turnera is CRITICAL: Connection refused by host [18:58:56] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [18:59:06] SMTP on turnera is CRITICAL: Connection refused [18:59:06] Free Memory on damiana is OK: OK - 7.6% (634156 kB) free. [18:59:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:59:07] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:59:26] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:59:37] Environment IPMI on turnera is CRITICAL: Connection refused by host [18:59:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:59:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:59:47] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [18:59:47] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [18:59:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [19:00:36] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [19:03:56] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [19:06:06] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (344528 kB) free! [19:17:27] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61705 MB (10% inode=99%): [19:17:57] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [19:23:46] Free Memory on turnera is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:23:47] Load avg. on turnera is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:24:16] ts-array5 on turnera is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:24:17] / on turnera is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:24:32] /tmp on turnera is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:24:32] DiskSuite on turnera is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [19:24:37] Environment IPMI on turnera is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:39:47] Free Memory on turnera is CRITICAL: Connection refused by host [19:39:47] Load avg. on turnera is CRITICAL: NRPE: Command check_load not defined [19:40:17] / on turnera is CRITICAL: NRPE: Command check_root not defined [19:40:17] ts-array5 on turnera is CRITICAL: NRPE: Command check_multipath not defined [19:40:27] /tmp on turnera is CRITICAL: NRPE: Command check_tmp not defined [19:40:27] DiskSuite on turnera is CRITICAL: NRPE: Command check_disksuite not defined [19:40:37] Environment IPMI on turnera is CRITICAL: NRPE: Command check_ipmi not defined [19:54:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1933329.000000 [19:54:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2918808.000000 [19:54:12] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [19:54:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:54:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:54:17] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:17] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:54:17] / on thyme is UNKNOWN: NRPE: Unable to read output [19:54:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1571681.000000 [19:54:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:54:18] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:54:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180276 [19:54:27] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:54:27] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:54:27] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:54:27] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:37] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [19:54:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:54:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:54:55] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [19:54:56] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 750602.000000 [19:54:56] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [19:54:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:54:56] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 945768.000000 [19:54:57] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:57] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:54:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:55:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:55:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180196.000000 [19:55:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [19:55:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1816121.000000 [19:55:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:55:16] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:55:17] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:55:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:17] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:26] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:26] SSH on mayapple is CRITICAL: Server answer: [19:55:27] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [19:55:27] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:55:36] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:55:37] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:55:37] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:55:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239680 MB (4% inode=63%): [19:55:37] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:14:50] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 81734 MB (8% inode=99%): [20:14:51] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [20:14:52] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [20:14:52] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:14:52] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:15:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 605987.000000 [20:15:40] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:15:41] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [20:15:42] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [20:15:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:15:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:15:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:15:51] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:15:51] / on thyme is UNKNOWN: NRPE: Unable to read output [20:15:52] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [20:15:52] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:09] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:16:40] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [20:16:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:16:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:17:40] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:17:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:23:50] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [20:23:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61555 MB (10% inode=99%): [20:34:40] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2163.000000 [20:34:41] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2164.000000 [20:40:40] Free Memory on turnera is CRITICAL: NRPE: Command check_free_mem not defined [20:40:40] /tmp on turnera is CRITICAL: NRPE: Command check_tmp not defined [20:40:40] Load avg. on turnera is CRITICAL: NRPE: Command check_load not defined [20:40:41] ts-array5 on turnera is CRITICAL: NRPE: Command check_multipath not defined [20:40:41] DiskSuite on turnera is CRITICAL: NRPE: Command check_disksuite not defined [20:40:42] Environment IPMI on turnera is CRITICAL: NRPE: Command check_ipmi not defined [20:40:42] / on turnera is CRITICAL: NRPE: Command check_root not defined [20:54:30] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:50] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [20:54:51] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 176999 [20:54:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:54:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1574905.000000 [20:54:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2922452.000000 [20:54:51] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1936621.000000 [20:54:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:55:00] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:00] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:00] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:55:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:29] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:30] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:39] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:55:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239524 MB (4% inode=63%): [20:55:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [20:55:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 949196.000000 [20:55:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177015.000000 [20:55:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 754029.000000 [20:55:41] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:55:50] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:51] SSH on mayapple is CRITICAL: Server answer: [20:55:51] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1819767.000000 [20:56:00] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:00] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:00] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:01] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:01] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:56:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:56:29] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:56:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:58:40] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3604.000000 [21:03:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3832.000000 [21:05:40] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3593.000000 [21:06:40] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3574.000000 [21:07:40] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [21:07:40] SMTP on turnera is CRITICAL: Connection refused [21:07:41] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [21:07:41] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [21:15:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 609024.000000 [21:15:42] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:15:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:15:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:15:52] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [21:15:53] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [21:15:54] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [21:15:55] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:15:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:16:00] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:10] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:40] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [21:16:40] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [21:16:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:16:51] / on thyme is UNKNOWN: NRPE: Unable to read output [21:16:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:17:09] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [21:17:40] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:17:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:17:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:17:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:22:40] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1677.000000 [21:23:50] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [21:23:50] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61371 MB (10% inode=99%): [21:30:40] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1765.000000 [21:40:40] Free Memory on turnera is CRITICAL: NRPE: Command check_free_mem not defined [21:40:40] /tmp on turnera is CRITICAL: NRPE: Command check_tmp not defined [21:41:40] ts-array5 on turnera is CRITICAL: NRPE: Command check_multipath not defined [21:41:41] Load avg. on turnera is CRITICAL: NRPE: Command check_load not defined [21:41:41] DiskSuite on turnera is CRITICAL: NRPE: Command check_disksuite not defined [21:41:41] / on turnera is CRITICAL: NRPE: Command check_root not defined [21:41:42] Environment IPMI on turnera is CRITICAL: NRPE: Command check_ipmi not defined [21:54:30] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:54:50] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [21:54:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1573676.000000 [21:54:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:54:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2926053.000000 [21:54:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1935778.000000 [21:54:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 177750 [21:55:00] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:55:00] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:00] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:09] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:55:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:29] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:30] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:40] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:55:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 754264.000000 [21:55:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 950262.000000 [21:55:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [21:55:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177759.000000 [21:55:42] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239413 MB (4% inode=63%): [21:55:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:55:50] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:51] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:51] SSH on mayapple is CRITICAL: Server answer: [21:55:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1823369.000000 [21:56:00] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:00] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:00] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:01] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:56:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:56:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:56:30] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:56:59] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:50] Free Memory on damiana is WARNING: WARNING - 6.1% (514236 kB) free! [22:02:50] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (389808 kB) free! [22:04:50] Free Memory on damiana is WARNING: WARNING - 5.5% (464888 kB) free! [22:07:40] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [22:07:40] SMTP on turnera is CRITICAL: Connection refused [22:07:41] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [22:07:41] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [22:15:51] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:15:52] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [22:15:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (410648 kB) free! [22:16:00] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:10] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:16:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:16:41] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [22:16:41] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:16:41] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [22:16:50] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [22:16:51] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [22:16:51] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:16:51] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:17:10] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [22:17:40] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:17:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:17:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:17:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 609441.000000 [22:17:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:17:51] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:17:51] / on thyme is UNKNOWN: NRPE: Unable to read output [22:17:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:19:50] nacht ts [22:23:50] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [22:23:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61217 MB (10% inode=99%): [22:40:40] /tmp on turnera is CRITICAL: NRPE: Command check_tmp not defined [22:40:41] Free Memory on turnera is CRITICAL: NRPE: Command check_free_mem not defined [22:41:40] ts-array5 on turnera is CRITICAL: NRPE: Command check_multipath not defined [22:41:41] Load avg. on turnera is CRITICAL: NRPE: Command check_load not defined [22:41:41] Environment IPMI on turnera is CRITICAL: NRPE: Command check_ipmi not defined [22:41:41] DiskSuite on turnera is CRITICAL: NRPE: Command check_disksuite not defined [22:41:42] / on turnera is CRITICAL: NRPE: Command check_root not defined [22:43:59] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [22:45:00] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [22:45:01] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [22:45:51] RAID on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:51] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:51] /sql on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:51] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:51] /tmp on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:52] / on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:52] Environment IPMI on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:52] Load avg. on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:53] / on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:53] /tmp on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:53] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:54] Load avg. on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:54] /tmp on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:55] / on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:55] Load avg. on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:46:59] Load avg. on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:47:00] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:47:29] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2406 [22:47:40] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2463 [22:47:51] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 237355 MB (24% inode=99%): [22:47:52] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 70162 MB (11% inode=99%): [22:47:52] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 5055 MB (99% inode=99%): [22:47:52] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 330799 MB (41% inode=99%): [22:47:52] / on z-dat-s4-a is OK: DISK OK - free space: / 6482 MB (21% inode=84%): [22:47:52] / on z-dat-s3-a is OK: DISK OK - free space: / 6482 MB (21% inode=84%): [22:47:53] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 5057 MB (99% inode=99%): [22:47:53] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 170970 MB (24% inode=99%): [22:47:53] / on z-dat-s6-a is OK: DISK OK - free space: / 6482 MB (21% inode=84%): [22:47:54] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:47:54] Load avg. on hyacinth is OK: OK - load average: 0.20, 0.22, 0.52 [22:47:55] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 5049 MB (99% inode=99%): [22:47:55] Load avg. on z-dat-s3-a is OK: OK - load average: 0.20, 0.22, 0.52 [22:47:56] Load avg. on z-dat-s7-a is OK: OK - load average: 0.20, 0.22, 0.52 [22:47:56] Load avg. on z-dat-s6-a is OK: OK - load average: 0.22, 0.23, 0.52 [22:48:00] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [22:48:00] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:48:40] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [22:52:29] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [22:52:30] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [22:54:29] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:54:51] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [22:54:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2929654.000000 [22:54:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 177002 [22:54:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1572117.000000 [22:54:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:54:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1935241.000000 [22:55:00] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [22:55:01] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:55:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:05] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:55:11] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:11] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:29] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:30] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:40] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:55:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [22:55:50] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:51] SSH on mayapple is CRITICAL: Server answer: [22:55:51] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1826969.000000 [22:55:59] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:00] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:00] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:01] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:09] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:19] wikidata replag on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [22:56:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [22:56:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:30] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:30] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 239379 MB (4% inode=63%): [22:56:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177033.000000 [22:56:41] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:56:50] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:51] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:51] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:52] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:52] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:00] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:00] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:01] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:01] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:10] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:10] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:10] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:10] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:19] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:20] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:20] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:20] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:20] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:20] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:21] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:21] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:22] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:22] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:23] Load avg. on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:23] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:24] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:24] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:25] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:25] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:29] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2329 [22:57:40] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:57:40] /tmp on hyacinth is OK: DISK OK - free space: /tmp 941 MB (99% inode=99%): [22:57:41] / on hyacinth is OK: DISK OK - free space: / 6480 MB (21% inode=84%): [22:57:41] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:57:41] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:57:42] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 948 MB (99% inode=99%): [22:57:42] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:57:42] Load avg. on z-dat-s4-a is OK: OK - load average: 0.21, 0.31, 0.53 [22:57:50] SMTP on z-dat-s7-a is OK: SMTP OK - 0.003 sec. response time [22:57:50] SMTP on z-dat-s3-a is OK: SMTP OK - 0.011 sec. response time [23:07:40] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [23:07:41] SMTP on turnera is CRITICAL: Connection refused [23:07:42] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [23:07:42] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [23:15:59] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:16:10] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:16:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:16:41] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:16:41] APT on yucca is WARNING: APT WARNING: 40 packages available for upgrade (0 critical updates). [23:16:42] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [23:16:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:16:51] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [23:16:51] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [23:16:51] APT on sage is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [23:16:52] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:16:52] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:17:19] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4225 [23:17:39] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [23:17:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:17:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:17:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 609304.000000 [23:17:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:17:51] / on thyme is UNKNOWN: NRPE: Unable to read output [23:17:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:18:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4201 [23:18:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:23:50] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [23:23:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61110 MB (10% inode=99%): [23:31:29] MySQL slave on z-dat-s7-a is OK: Uptime: 2614 Threads: 2 Questions: 17 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 15 Queries per second avg: 0.6 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [23:31:39] MySQL on z-dat-s7-a is OK: Uptime: 2625 Threads: 1 Questions: 150 Slow queries: 0 Opens: 53 Flush tables: 1 Open tables: 44 Queries per second avg: 0.57 [23:38:29] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3739 [23:39:29] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3507 [23:40:40] /tmp on turnera is CRITICAL: NRPE: Command check_tmp not defined [23:41:40] ts-array5 on turnera is CRITICAL: NRPE: Command check_multipath not defined [23:41:41] Free Memory on turnera is CRITICAL: NRPE: Command check_free_mem not defined [23:42:40] Load avg. on turnera is CRITICAL: NRPE: Command check_load not defined [23:42:41] DiskSuite on turnera is CRITICAL: NRPE: Command check_disksuite not defined [23:42:41] / on turnera is CRITICAL: NRPE: Command check_root not defined [23:42:41] Environment IPMI on turnera is CRITICAL: NRPE: Command check_ipmi not defined [23:54:30] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:54:50] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [23:55:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:55:29] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:55:29] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:55:40] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:55:40] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [23:55:50] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:50] SSH on mayapple is CRITICAL: Server answer: [23:55:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:55:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 175200 [23:55:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1934486.000000 [23:55:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1570771.000000 [23:55:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2933314.000000 [23:55:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:56:00] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:00] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:01] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:01] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:01] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:01] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:01] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 760936.000000 [23:56:30] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:30] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:30] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 956951.000000 [23:56:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 238519 MB (4% inode=63%): [23:56:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 175123.000000 [23:56:41] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:56:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1830629.000000 [23:57:29] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3339 [23:57:50] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:58:00] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds