[00:02:43] Load avg. on nightshade is OK: OK - load average: 2.10, 3.74, 14.71 [00:03:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [00:04:43] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [00:11:43] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3016.000000 [00:13:44] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1680.000000 [00:15:43] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3088 [00:15:43] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3770 [00:15:53] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4004 [00:17:43] MySQL slave on z-dat-s4-a is OK: Uptime: 1053613 Threads: 1 Questions: 42885600 Slow queries: 12 Opens: 79 Flush tables: 1 Open tables: 72 Queries per second avg: 40.703 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1437 [00:17:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3107 [00:17:43] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3587 [00:17:43] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3434.000000 [00:17:53] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3412 [00:24:43] MySQL slave on z-dat-s2-b is OK: Uptime: 147156 Threads: 18 Questions: 80593230 Slow queries: 3477 Opens: 899484 Flush tables: 1 Open tables: 256 Queries per second avg: 547.672 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1716 [00:25:42] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1764.000000 [00:25:53] MySQL slave on daphne is OK: Uptime: 2349553 Threads: 132 Questions: 269813388 Slow queries: 498753 Opens: 85729 Flush tables: 1 Open tables: 1927 Queries per second avg: 114.836 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1722 [00:25:53] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 82052 MB (8% inode=99%): [00:30:43] MySQL slave on z-dat-s1-b is OK: Uptime: 173137 Threads: 15 Questions: 108571856 Slow queries: 4619 Opens: 1144 Flush tables: 1 Open tables: 521 Queries per second avg: 627.86 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1791 [00:44:22] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:44:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:44:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:44:44] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:44:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741327.000000 [00:44:45] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46190.000000 [00:44:46] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 926487.000000 [00:44:46] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [00:44:47] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:44:47] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:44:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [00:44:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 185468.000000 [00:44:51] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:44:54] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:44:55] SSH on mayapple is CRITICAL: Server answer: [00:44:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:44:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:45:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:33] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 246032 MB (4% inode=63%): [00:45:44] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [00:45:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:45:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1660760.000000 [00:45:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:45:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2763508.000000 [00:45:46] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1815433.000000 [00:45:46] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [00:45:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:45:47] / on thyme is UNKNOWN: NRPE: Unable to read output [00:45:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1590450.000000 [00:45:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:45:49] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:45:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:46:03] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:46:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:04] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:04] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:05] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:06] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:07] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:14] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:14] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:15] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:34] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:46:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:20] my mail? [01:08:30] what? [01:11:40] closedmouth: DaBPunkt is offline. [01:12:03] weird, i didn't send any mail [01:20:36] closedmouth: Well, who knows :-). [01:44:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:44:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:45:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:45:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:45:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743512.000000 [01:45:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245969 MB (4% inode=63%): [01:45:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [01:45:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928187.000000 [01:45:44] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45468.000000 [01:45:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:45:45] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [01:45:46] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:45:46] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:45:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [01:46:02] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:46:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:13] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1664420.000000 [01:46:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2767168.000000 [01:46:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:46:43] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [01:46:43] / on thyme is UNKNOWN: NRPE: Unable to read output [01:46:44] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:46:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:46:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1593727.000000 [01:46:45] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1819034.000000 [01:46:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:46:46] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:46:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:03] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:03] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:04] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:12] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:13] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:13] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:19:38] [[Special:Log/newusers]] create 10 * Cualo1 * (New user account) [02:44:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:44:44] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:45:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:33] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:45:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:45:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743105.000000 [02:45:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245926 MB (4% inode=63%): [02:45:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [02:45:44] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36800.000000 [02:45:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 926782.000000 [02:45:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:45:45] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [02:45:46] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:45:46] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:45:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 192728.000000 [02:46:02] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:46:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:46:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1668020.000000 [02:46:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2770767.000000 [02:46:44] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [02:46:44] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:46:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:46:44] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1593673.000000 [02:46:44] / on thyme is UNKNOWN: NRPE: Unable to read output [02:46:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:46:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:46:45] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:46:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:03] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:03] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:04] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:13] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:13] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:13] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1822564.000000 [03:44:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:44:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:45:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:45:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:45:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:45:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:45:53] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:45:53] SSH on mayapple is CRITICAL: Server answer: [03:45:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:45:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:45:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:46:02] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:46:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:46:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 742107.000000 [03:46:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245887 MB (4% inode=63%): [03:46:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [03:46:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 925762.000000 [03:46:44] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30606.000000 [03:46:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:46:44] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [03:46:45] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:46:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:46:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 196388.000000 [03:46:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [03:46:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1671620.000000 [03:47:02] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:03] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:13] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:13] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:44] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1825908.000000 [04:44:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:44:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:45:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:45:44] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:45:53] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:45:53] SSH on mayapple is CRITICAL: Server answer: [04:45:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:45:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:45:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:46:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:46:13] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245832 MB (4% inode=63%): [04:46:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741132.000000 [04:46:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [04:46:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:46:53] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:47:02] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:03] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:12] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:12] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:33] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 925785.000000 [04:47:44] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22929.000000 [04:47:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:47:44] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [04:47:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:47:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:47:44] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1675280.000000 [04:47:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [04:47:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 200048.000000 [04:47:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2778028.000000 [04:47:46] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1829389.000000 [04:47:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:44:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:44:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:45:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:45:44] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:45:53] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [05:45:54] SSH on mayapple is CRITICAL: Server answer: [05:45:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:45:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:45:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:46:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:13] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:46:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 740693.000000 [05:46:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245803 MB (4% inode=63%): [05:46:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [05:46:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:02] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:47:03] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:04] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:12] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:12] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:33] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:44] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15125.000000 [05:47:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 926222.000000 [05:47:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:47:44] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [05:47:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:47:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:47:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [05:47:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1678880.000000 [05:47:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 203648.000000 [05:47:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2781628.000000 [05:47:46] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [05:47:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:44:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:44:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:45:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:33] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:45:53] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:45:53] SSH on mayapple is CRITICAL: Server answer: [06:45:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:45:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:45:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:46:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:46:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:46:32] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:46:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741688.000000 [06:46:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245614 MB (4% inode=63%): [06:46:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [06:46:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:03] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:03] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:03] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:13] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:13] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:13] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:33] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:43] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8287.000000 [06:47:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 927436.000000 [06:47:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:47:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [06:47:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:47:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [06:47:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 206372.000000 [06:47:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1682480.000000 [06:47:45] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:47:46] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [06:47:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:47:47] / on thyme is UNKNOWN: NRPE: Unable to read output [07:11:53] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1724.000000 [07:16:43] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3564.000000 [07:29:43] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1683.000000 [07:44:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:44:44] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:45:23] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:45:23] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:45:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:45:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:45:53] SSH on mayapple is CRITICAL: Server answer: [07:45:53] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:45:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:45:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:45:57] [[Special:Log/newusers]] create 10 * Sigrid89 * (New user account) [07:46:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:46:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 742979.000000 [07:46:44] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [07:46:44] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245541 MB (4% inode=63%): [07:46:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:46:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:47:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:03] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:03] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:13] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:13] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:22] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:23] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:23] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928751.000000 [07:47:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:47:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:47:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [07:47:44] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1686080.000000 [07:47:44] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [07:47:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207264.000000 [07:47:45] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:47:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:47:46] / on thyme is UNKNOWN: NRPE: Unable to read output [07:47:46] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [07:47:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1598180.000000 [08:44:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:45:22] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:45:22] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:45:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:45:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:45:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:45:53] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:45:54] SSH on mayapple is CRITICAL: Server answer: [08:45:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:45:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:46:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:23] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:46:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:47:03] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:12] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:12] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:13] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:13] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:22] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:23] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:23] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:33] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 742909.000000 [08:47:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245404 MB (4% inode=63%): [08:47:44] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [08:47:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207480.000000 [08:47:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 207480 [08:47:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2792432.000000 [08:47:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:47:53] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:47:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1842631.000000 [08:48:03] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:48:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928788.000000 [08:48:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:48:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:48:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [08:48:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1689741.000000 [08:48:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:48:44] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [08:48:45] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [08:48:45] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:48:46] / on thyme is UNKNOWN: NRPE: Unable to read output [08:48:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:48:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1596553.000000 [09:44:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:45:23] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:23] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:33] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:45:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:46:05] SSH on mayapple is CRITICAL: Server answer: [09:46:05] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:46:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:46:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:46:22] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:56] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:47:11] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:47:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:17] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:17] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:17] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:17] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:33] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:33] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:33] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:33] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743428.000000 [09:47:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245178 MB (4% inode=63%): [09:47:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [09:47:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207888.000000 [09:47:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 207884 [09:47:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:47:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2796038.000000 [09:47:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1846089.000000 [09:47:54] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:48:04] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:48:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928532.000000 [09:48:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:48:53] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [09:48:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:48:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:48:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:48:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1693344.000000 [09:48:54] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [09:48:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [09:48:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:48:56] / on thyme is UNKNOWN: NRPE: Unable to read output [09:48:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1595465.000000 [09:48:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:44:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:45:32] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:45:33] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:45:33] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:22] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:32] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:42] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:46:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:47:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:04] SSH on mayapple is CRITICAL: Server answer: [10:47:04] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:47:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:47:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:47:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:13] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:13] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:13] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:13] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:14] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:33] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:33] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:33] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743734.000000 [10:47:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:47:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [10:47:53] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 245091 MB (4% inode=63%): [10:47:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208415.000000 [10:47:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 208414 [10:47:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2799638.000000 [10:47:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:47:54] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:47:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1849565.000000 [10:48:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:48:03] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:48:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928266.000000 [10:48:53] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [10:49:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:49:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:49:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:49:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [10:49:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:49:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1697005.000000 [10:49:54] / on thyme is UNKNOWN: NRPE: Unable to read output [10:49:54] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [10:49:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1595180.000000 [10:49:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:49:56] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:44:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:44:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:45:33] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:45:33] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:45:33] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:22] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:46:43] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:47:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:03] SSH on mayapple is CRITICAL: Server answer: [11:47:03] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:47:03] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:47:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:47:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:13] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:13] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:13] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:13] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:14] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:15] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:32] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:32] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743889.000000 [11:47:43] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:47:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [11:47:53] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244888 MB (4% inode=63%): [11:47:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208723.000000 [11:47:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 208721 [11:47:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:47:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2803238.000000 [11:47:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1853070.000000 [11:47:54] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:48:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:48:04] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:48:33] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:48:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:48:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928240.000000 [11:48:53] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [11:49:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:49:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:49:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:49:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [11:49:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:49:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1700605.000000 [11:50:53] / on thyme is UNKNOWN: NRPE: Unable to read output [11:50:53] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [11:50:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1592436.000000 [11:50:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:50:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:26:03] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 77594 MB (7% inode=99%): [12:33:54] Free Memory on damiana is CRITICAL: CRITICAL - 3.0% (251864 kB) free! [12:36:55] Free Memory on damiana is OK: OK - 12.0% (1005400 kB) free. [12:44:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:44:54] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:45:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:45:44] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:24] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:44] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:44] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:46:44] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:47:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:14] SSH on mayapple is CRITICAL: Server answer: [12:47:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [12:47:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:47:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:47:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:24] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:24] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:43] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:44] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744068.000000 [12:47:45] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:47:45] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [12:47:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 209553.000000 [12:47:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 209549 [12:47:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1856569.000000 [12:47:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2806839.000000 [12:47:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:47:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:48:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:48:14] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:48:44] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:48:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:48:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928488.000000 [12:48:44] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244785 MB (4% inode=63%): [12:49:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:49:45] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [12:49:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [12:49:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:49:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1704206.000000 [12:50:44] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:50:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:50:54] / on thyme is UNKNOWN: NRPE: Unable to read output [12:50:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1593781.000000 [12:50:54] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [12:50:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:50:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:54:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2152 [13:44:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:44:54] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:45:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:45:45] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:24] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:44] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:46:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:45] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:47:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:14] SSH on mayapple is CRITICAL: Server answer: [13:47:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [13:47:17] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:47:17] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:26] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:26] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:26] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:26] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:46] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:47:46] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744379.000000 [13:47:46] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [13:47:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 210381.000000 [13:47:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:47:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:47:56] Free Memory on damiana is CRITICAL: CRITICAL - 3.1% (260528 kB) free! [13:47:56] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210380 [13:47:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2810440.000000 [13:47:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1860048.000000 [13:48:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:48:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:48:44] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:48:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:48:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928946.000000 [13:48:53] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244629 MB (4% inode=63%): [13:49:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:49:45] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [13:49:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [13:49:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:49:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1707809.000000 [13:50:43] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:50:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:50:55] MySQL slave on z-dat-s2-b is OK: Uptime: 195522 Threads: 7 Questions: 117051647 Slow queries: 4233 Opens: 1069842 Flush tables: 1 Open tables: 256 Queries per second avg: 598.662 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1727 [13:50:56] / on thyme is UNKNOWN: NRPE: Unable to read output [13:50:57] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [13:50:57] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1595527.000000 [13:50:57] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:50:57] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:53:54] Free Memory on damiana is OK: OK - 26.8% (2249192 kB) free. [14:26:54] Free Memory on damiana is CRITICAL: CRITICAL - 2.7% (225988 kB) free! [14:27:50] hello all [14:31:54] Free Memory on damiana is OK: OK - 28.6% (2398280 kB) free. [14:44:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:44:54] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:45:43] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:46:24] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:46:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:46:44] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:46:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:13] SSH on mayapple is CRITICAL: Server answer: [14:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:47:44] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [14:47:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:47:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 210665.000000 [14:47:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:47:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [14:48:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [14:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:48:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:48:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:48:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [14:48:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:24] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:24] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744924.000000 [14:48:44] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [14:48:44] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:48:44] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [14:48:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210650 [14:48:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2814099.000000 [14:48:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1863614.000000 [14:49:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 929448.000000 [14:49:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:49:45] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244424 MB (4% inode=63%): [14:49:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:49:54] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [14:49:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [14:50:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:50:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:50:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1711468.000000 [14:50:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:51:54] / on thyme is UNKNOWN: NRPE: Unable to read output [14:51:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1594396.000000 [14:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [14:51:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:51:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:44:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:45:04] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:45:43] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:46:24] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:46:44] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:46:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:46:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:13] SSH on mayapple is CRITICAL: Server answer: [15:47:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:44] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:47:44] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:47:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 210330.000000 [15:48:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:48:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [15:48:24] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:24] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:24] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 745496.000000 [15:48:44] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:48:44] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [15:48:44] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:46] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210318 [15:49:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:49:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:49:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:49:14] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:49:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:49:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 930765.000000 [15:49:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:49:44] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 244143 MB (4% inode=63%): [15:49:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:49:54] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [15:49:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [15:49:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2817758.000000 [15:49:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1867153.000000 [15:50:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:51:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:51:54] / on thyme is UNKNOWN: NRPE: Unable to read output [15:51:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:51:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1715130.000000 [15:51:54] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [15:51:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:51:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:52:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1595265.000000 [16:18:54] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (411228 kB) free! [16:22:02] Free Memory on damiana is OK: OK - 28.6% (2395572 kB) free. [16:44:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:45:04] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:46:24] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:46:44] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:46:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:46:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:13] SSH on mayapple is CRITICAL: Server answer: [16:47:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:44] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:47:45] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:47:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 210260.000000 [16:48:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:48:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [16:48:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:24] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:24] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:24] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:44] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:48:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744723.000000 [16:48:45] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [16:48:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:46] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:46] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:46] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:54] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:48:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:48:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210240 [16:49:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:49:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:49:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:49:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:49:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931651.000000 [16:49:46] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243916 MB (4% inode=63%): [16:49:46] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:49:54] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [16:49:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [16:49:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1870664.000000 [16:49:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2821361.000000 [16:50:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:51:54] / on thyme is UNKNOWN: NRPE: Unable to read output [16:51:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [16:51:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:51:56] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:51:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1718732.000000 [16:51:56] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:52:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1594560.000000 [16:55:00] Free Memory on damiana is CRITICAL: CRITICAL - 3.7% (310944 kB) free! [16:57:54] Free Memory on damiana is OK: OK - 26.8% (2241884 kB) free. [17:31:54] Free Memory on damiana is WARNING: WARNING - 5.4% (449076 kB) free! [17:32:55] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (399144 kB) free! [17:34:53] Free Memory on damiana is OK: OK - 28.5% (2385384 kB) free. [17:44:44] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:45:54] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:46:24] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:46:44] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:46:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:13] SSH on mayapple is CRITICAL: Server answer: [17:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:44] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:47:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 209302.000000 [17:48:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:48:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [17:48:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:24] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:24] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:44] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:48:44] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:48:54] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:49:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:24] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744275.000000 [17:49:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:49:44] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [17:49:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:44] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:46] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:54] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [17:49:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 209247 [17:49:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:50:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:50:13] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:50:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931647.000000 [17:50:44] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243741 MB (4% inode=63%): [17:50:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:50:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:50:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [17:50:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2825019.000000 [17:50:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1874242.000000 [17:51:54] / on thyme is UNKNOWN: NRPE: Unable to read output [17:51:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [17:52:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:52:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1594309.000000 [17:52:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1722390.000000 [17:52:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:52:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:05:54] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (339488 kB) free! [18:10:54] Free Memory on damiana is OK: OK - 27.4% (2296896 kB) free. [18:21:14] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.373 second response time [18:22:14] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.173 second response time [18:28:23] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:28:24] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:28:44] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:30:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:30:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:30:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:31:07] @replag [18:33:25] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:44:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:46:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:47:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:16] SSH on mayapple is CRITICAL: Server answer: [18:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:47:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208323.000000 [18:48:14] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [18:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [18:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [18:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:49:26] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:26] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:49:56] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744324.000000 [18:49:56] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [18:49:56] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [18:50:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 208377 [18:50:18] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:50:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243688 MB (4% inode=63%): [18:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931753.000000 [18:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:50:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [18:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1877748.000000 [18:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2828625.000000 [18:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [18:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [18:52:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1591673.000000 [18:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:52:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:52:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1725996.000000 [18:53:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:31:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:45:51] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:46:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:46:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:47:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:15] SSH on mayapple is CRITICAL: Server answer: [19:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:47:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:47:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208222.000000 [19:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [19:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:26] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:44] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:44] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 745175.000000 [19:49:56] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [19:49:56] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [19:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 208222 [19:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:50:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243443 MB (4% inode=63%): [19:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 932962.000000 [19:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:50:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [19:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2832224.000000 [19:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1881186.000000 [19:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [19:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [19:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:52:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1591708.000000 [19:52:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:52:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1729596.000000 [19:53:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:53:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:06:15] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.048 second response time [20:08:15] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [20:25:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [20:31:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:31:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:44:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [20:45:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [20:45:15] SMTP on turnera is CRITICAL: Connection refused [20:45:15] SSH on turnera is CRITICAL: Connection refused [20:45:16] DiskSuite on turnera is CRITICAL: Connection refused by host [20:45:16] /tmp on turnera is CRITICAL: Connection refused by host [20:45:16] ts-array5 on turnera is CRITICAL: Connection refused by host [20:45:16] / on turnera is CRITICAL: Connection refused by host [20:45:16] Free Memory on turnera is CRITICAL: Connection refused by host [20:45:17] Load avg. on turnera is CRITICAL: Connection refused by host [20:45:45] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:46:34] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:46:44] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:46:44] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:47:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:14] SSH on mayapple is CRITICAL: Server answer: [20:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:45] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:47:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207815.000000 [20:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:48:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:31] @replag [20:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:49:04] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:49:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:49:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744638.000000 [20:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [20:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [20:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 207800 [20:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:51:16] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:51:27] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 934737.000000 [20:51:27] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243229 MB (4% inode=63%): [20:51:27] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:51:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [20:51:27] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1884646.000000 [20:51:28] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2835824.000000 [20:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [20:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [20:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:52:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1591598.000000 [20:52:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:52:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1733196.000000 [20:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:53:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:31:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:46:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:47:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:15] SSH on mayapple is CRITICAL: Server answer: [21:47:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:47:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:47:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208190.000000 [21:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:49:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:24] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744479.000000 [21:49:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:49:56] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [21:49:56] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [21:50:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 208179 [21:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:50:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 935364.000000 [21:50:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 243079 MB (4% inode=63%): [21:50:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:50:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [21:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2839425.000000 [21:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1888156.000000 [21:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [21:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [21:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:52:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1591326.000000 [21:52:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:52:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1736796.000000 [21:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:54:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [22:06:16] Free Memory on turnera is CRITICAL: Connection refused by host [22:06:16] ts-array5 on turnera is CRITICAL: Connection refused by host [22:06:16] / on turnera is CRITICAL: Connection refused by host [22:06:16] /tmp on turnera is CRITICAL: Connection refused by host [22:06:17] DiskSuite on turnera is CRITICAL: Connection refused by host [22:06:55] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [22:07:05] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [22:07:05] Environment IPMI on turnera is CRITICAL: Connection refused by host [22:07:15] SSH on turnera is CRITICAL: Connection refused [22:07:15] SMTP on turnera is CRITICAL: Connection refused [22:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:46:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:47:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:47:16] SSH on mayapple is CRITICAL: Server answer: [22:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:47:44] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:47:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:47:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208268.000000 [22:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:48:58] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:48:58] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:49:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:49:24] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:24] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:44] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 744432.000000 [22:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [22:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [22:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 208323 [22:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:50:45] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:50:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 935658.000000 [22:50:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 242974 MB (4% inode=63%): [22:50:56] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:50:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [22:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2843024.000000 [22:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1891689.000000 [22:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [22:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [22:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:52:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1590866.000000 [22:52:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:52:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1740396.000000 [22:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:54:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:58:15] SSH on turnera is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [23:06:15] Load avg. on turnera is CRITICAL: Connection refused by host [23:06:54] NTP on turnera is CRITICAL: NTP CRITICAL: No response from NTP server [23:07:04] FMA on turnera is CRITICAL: ERROR - unexpected output from snmpwalk [23:07:06] Environment IPMI on turnera is CRITICAL: Connection refused by host [23:07:15] SMTP on turnera is CRITICAL: Connection refused [23:07:15] /tmp on turnera is CRITICAL: Connection refused by host [23:07:15] ts-array5 on turnera is CRITICAL: Connection refused by host [23:07:15] / on turnera is CRITICAL: Connection refused by host [23:07:15] DiskSuite on turnera is CRITICAL: Connection refused by host [23:07:15] Free Memory on turnera is CRITICAL: Connection refused by host [23:31:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:31:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:44:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:46:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:45] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:45] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:47:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:14] SSH on mayapple is CRITICAL: Server answer: [23:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:45] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:00] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:48:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 209542.000000 [23:48:15] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:48:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:25] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:48:55] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:49:04] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:49:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:49:25] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:25] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:45] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:45] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:45] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:45] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:49:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743433.000000 [23:49:55] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [23:49:55] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [23:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 209523 [23:50:15] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:50:44] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:50:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 242096 MB (4% inode=63%): [23:50:56] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 938986.000000 [23:50:56] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:50:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [23:51:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1895209.000000 [23:51:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2846625.000000 [23:51:55] / on thyme is UNKNOWN: NRPE: Unable to read output [23:51:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [23:52:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:52:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1591313.000000 [23:52:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:52:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1743996.000000 [23:53:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:55:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1