[00:11:05] @replag [00:12:12] * russblau didn't really think that would work [00:17:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:17:38] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:41:58] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:42:48] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:42:48] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:43:38] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:43:38] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:44:08] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:44:08] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:48:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 649942.000000 [00:48:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263659 MB (4% inode=64%): [00:48:18] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [00:48:19] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:48:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:48:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [00:48:29] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [00:48:29] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 204842.000000 [00:48:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:48:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2516132.000000 [00:48:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1650602.000000 [00:48:38] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:38] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [00:48:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:48:48] / on thyme is UNKNOWN: NRPE: Unable to read output [00:48:48] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1480693.000000 [00:48:48] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:48:49] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:48:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:48:58] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:58] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:48:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:48:58] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:48:58] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:58] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:48:59] SSH on mayapple is CRITICAL: Server answer: [00:48:59] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:48:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:49:00] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:49:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:49:18] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:49:18] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [00:49:18] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:18] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [00:49:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 829901.000000 [00:49:19] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17836.000000 [00:49:28] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:28] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24329.000000 [00:49:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1401786.000000 [00:49:48] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:48] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:57] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:54:18] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:59:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:59:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:59:28] aliasd on nightshade is CRITICAL: Connection refused [01:17:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:17:38] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:32:18] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3540.000000 [01:33:18] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1413.000000 [01:36:18] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1369.000000 [01:41:58] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [01:42:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:42:49] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:43:38] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:43:38] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:44:08] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:44:09] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:48:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 651185.000000 [01:48:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263594 MB (4% inode=64%): [01:48:18] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [01:48:18] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:48:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:48:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [01:48:28] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [01:48:28] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208442.000000 [01:48:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:48:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2519023.000000 [01:48:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1654076.000000 [01:48:38] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:38] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [01:48:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:48] / on thyme is UNKNOWN: NRPE: Unable to read output [01:48:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:48:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1479595.000000 [01:48:48] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:48] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:48:49] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:48:50] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:58] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:48:58] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:48:58] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:48:58] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:59] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:48:59] SSH on mayapple is CRITICAL: Server answer: [01:49:00] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:49:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:49:01] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:49:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:49:18] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:49:18] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [01:49:18] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:18] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [01:49:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 831104.000000 [01:49:28] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:28] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27930.000000 [01:49:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1405385.000000 [01:49:48] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:49] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:49:58] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:54:18] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [01:59:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:59:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:59:28] aliasd on nightshade is CRITICAL: Connection refused [02:17:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:17:38] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:41:58] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:54:18] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [02:59:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:59:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:59:28] aliasd on nightshade is CRITICAL: Connection refused [03:10:18] MySQL on z-dat-s1-b is OK: Uptime: 30176 Threads: 10 Questions: 63 Slow queries: 0 Opens: 31 Flush tables: 1 Open tables: 24 Queries per second avg: 0.2 [03:17:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:17:38] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:30:27] aliasd on nightshade is OK: TCP OK - 0.004 second response time on port 984 [500 Not found.] [03:41:58] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:42:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:42:49] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:43:38] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:43:38] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:44:07] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:44:07] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:48:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 653196.000000 [03:48:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263378 MB (4% inode=64%): [03:48:18] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [03:48:28] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:48:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:48:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [03:48:28] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 142102 [03:48:28] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 215642.000000 [03:48:28] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:48:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2524590.000000 [03:48:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1660993.000000 [03:48:38] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [03:48:38] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:48] / on thyme is UNKNOWN: NRPE: Unable to read output [03:48:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:48:48] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1474931.000000 [03:48:48] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:49] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:48:50] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:48:58] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:48:58] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:48:58] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:48:58] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:59] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:48:59] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:49:00] SSH on mayapple is CRITICAL: Server answer: [03:49:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:49:01] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [03:49:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:49:18] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:49:18] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:18] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [03:49:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 830917.000000 [03:49:28] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:28] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35130.000000 [03:49:29] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1412585.000000 [03:49:48] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:48] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:49:58] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:54:18] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [03:59:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:59:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:17:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:17:38] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:41:58] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:42:48] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:42:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:43:38] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:43:38] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:44:08] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:44:08] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:48:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263329 MB (4% inode=64%): [04:48:18] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [04:48:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 653145.000000 [04:48:28] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:48:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:48:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [04:48:28] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134280 [04:48:28] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 219242.000000 [04:48:28] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:48:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2527628.000000 [04:48:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1664411.000000 [04:48:38] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [04:48:38] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:48] / on thyme is UNKNOWN: NRPE: Unable to read output [04:48:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:48:48] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1469958.000000 [04:48:48] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:48] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:49] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:48:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:48:58] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:48:58] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:48:58] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:48:58] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:59] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:48:59] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:49:00] SSH on mayapple is CRITICAL: Server answer: [04:49:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:49:01] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [04:49:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:49:18] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:49:18] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:18] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [04:49:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 829174.000000 [04:49:28] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38729.000000 [04:49:28] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1416186.000000 [04:49:48] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:49] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:49:57] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:54:18] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [04:59:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:59:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:12:48] Free Memory on damiana is CRITICAL: CRITICAL - 3.9% (324156 kB) free! [05:17:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:17:46] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:25:37] Free Memory on damiana is WARNING: WARNING - 6.7% (563256 kB) free! [05:28:38] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (402200 kB) free! [05:38:18] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [05:40:48] SMF on ptolemy is CRITICAL: ERROR - maintenance: svc:/network/ts/apache22:default [05:40:58] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52741 MB (8% inode=99%): [05:41:28] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [05:41:58] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:42:48] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:42:48] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:43:37] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:43:38] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:44:08] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:44:08] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:48:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 652774.000000 [05:48:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263231 MB (4% inode=64%): [05:48:18] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [05:48:28] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:48:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:48:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [05:48:28] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 130420 [05:48:28] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 222037.000000 [05:48:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:48:37] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2530543.000000 [05:48:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1667448.000000 [05:48:39] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [05:48:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:48] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:49] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:49] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:48:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:48:49] / on thyme is UNKNOWN: NRPE: Unable to read output [05:48:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:48:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [05:48:58] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:48:58] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:48:59] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:48:59] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:48:59] SSH on mayapple is CRITICAL: Server answer: [05:48:59] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [05:48:59] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:48:59] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 222057 [05:49:00] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:49:18] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:49:19] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [05:49:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 828268.000000 [05:49:19] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:28] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42330.000000 [05:49:29] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:29] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1419786.000000 [05:49:48] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:48] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:49:58] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:54:19] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [05:59:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:59:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:02:38] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (335236 kB) free! [06:04:57] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:17:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:18:38] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:23:18] MySQL on z-dat-s2-b is OK: Uptime: 3079 Threads: 11 Questions: 458 Slow queries: 1 Opens: 121 Flush tables: 1 Open tables: 114 Queries per second avg: 0.148 [06:27:08] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 217654.000000 [06:27:08] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 496329.000000 [06:29:38] Free Memory on damiana is WARNING: WARNING - 5.1% (427876 kB) free! [06:31:38] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (397964 kB) free! [06:41:58] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:42:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:42:49] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:43:38] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:43:38] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:44:08] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:44:08] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:46:48] SMF on ptolemy is OK: OK - all services online [06:48:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263179 MB (4% inode=64%): [06:48:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 652659.000000 [06:48:19] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [06:48:28] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:48:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:48:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [06:48:28] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 126168 [06:48:28] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 221968.000000 [06:48:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:48:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2533710.000000 [06:48:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1670165.000000 [06:48:38] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [06:48:38] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:48] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:49] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:49] / on thyme is UNKNOWN: NRPE: Unable to read output [06:48:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:48:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:48:49] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:48:58] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:48:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:48:58] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:48:58] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:48:58] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:48:59] SSH on mayapple is CRITICAL: Server answer: [06:48:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:49:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:00] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 221970 [06:49:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:49:18] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:49:18] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [06:49:28] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45930.000000 [06:49:29] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:29] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1423385.000000 [06:49:48] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:48] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1469554.000000 [06:49:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:49:58] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:49:58] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 828281.000000 [06:50:18] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:54:18] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19548 [06:59:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:59:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:08:58] MySQL slave on z-dat-s5-b is OK: Uptime: 5803 Threads: 2 Questions: 4530845 Slow queries: 49 Opens: 452 Flush tables: 1 Open tables: 254 Queries per second avg: 780.776 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [07:17:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:18:38] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:29:50] RAID on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:43:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:43:28] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 78742 MB (8% inode=99%): [07:43:47] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:43:50] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [07:44:16] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:44:21] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:44:26] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:29] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:44:45] / on thyme is UNKNOWN: NRPE: Unable to read output [07:44:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:45:06] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:45:10] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:45:17] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:45:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:45:37] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [07:46:22] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:48:41] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:48:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [07:48:44] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263102 MB (4% inode=64%): [07:48:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 221121.000000 [07:48:54] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:04] SSH on mayapple is CRITICAL: Server answer: [07:49:04] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [07:49:05] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:49:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:49:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:49:34] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121247 [07:49:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 653296.000000 [07:49:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 495599.000000 [07:49:39] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 221146.000000 [07:49:41] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2536447.000000 [07:49:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49538.000000 [07:49:44] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:46] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:49:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:52] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:55] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:57] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:50:02] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:50:05] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:50:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:50:32] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1670980.000000 [07:50:34] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:42] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:50:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1470001.000000 [07:50:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1427053.000000 [07:50:57] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:02] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:10] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:34] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:51:53] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:51:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 829496.000000 [07:52:03] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:05:24] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:05:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:07:14] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:07:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:11:44] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [08:12:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52594 MB (8% inode=99%): [08:43:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:44:27] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:44:36] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:44:47] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [08:44:52] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:44:58] / on thyme is UNKNOWN: NRPE: Unable to read output [08:45:11] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:45:31] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:45:34] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:46:11] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:46:11] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [08:46:31] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:46:51] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:49:01] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:49:11] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:12] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:31] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:49:41] SSH on mayapple is CRITICAL: Server answer: [08:49:42] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 262913 MB (4% inode=64%): [08:49:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:49:52] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [08:50:01] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [08:50:04] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 223637.000000 [08:50:05] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:50:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:50:31] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114321 [08:50:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:32] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 653780.000000 [08:50:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2537714.000000 [08:50:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53197.000000 [08:50:34] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:41] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:51] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:52] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:51:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1469223.000000 [08:51:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1674257.000000 [08:51:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1430684.000000 [08:51:31] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:32] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:32] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:39] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:43] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:44] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:52:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:52:45] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:52:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 831465.000000 [09:05:38] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:06:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:07:44] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:07:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:12:35] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [09:12:35] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52534 MB (8% inode=99%): [09:22:05] [[Special:Log/newusers]] create 10 * Durifon * (New user account) [09:44:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:44:36] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:05] / on thyme is UNKNOWN: NRPE: Unable to read output [09:45:06] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:14] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:45:15] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [09:45:35] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:48] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:46:15] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [09:46:17] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:46:35] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:05] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:47:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:47:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:49:24] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:49:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:35] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:49:45] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:55] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:50:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:06] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:50:06] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:06] SSH on mayapple is CRITICAL: Server answer: [09:50:15] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 226171.000000 [09:50:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 262769 MB (4% inode=64%): [09:50:35] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106987 [09:50:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 654235.000000 [09:50:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2540426.000000 [09:50:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [09:50:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56822.000000 [09:50:56] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [09:51:05] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:07] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:16] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:16] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1465976.000000 [09:51:18] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1676711.000000 [09:51:25] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1434296.000000 [09:51:45] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:46] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:51:47] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:52:05] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:05] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:52:05] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:06] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:24] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:25] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:06] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 495102.000000 [09:53:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 831867.000000 [09:53:26] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 226318.000000 [09:53:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:04:44] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (376776 kB) free! [10:05:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:06:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:07:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:08:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:14:16] Free Memory on damiana is WARNING: WARNING - 5.1% (428740 kB) free! [10:16:15] Free Memory on damiana is CRITICAL: CRITICAL - 3.7% (308380 kB) free! [10:39:05] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [10:39:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52380 MB (8% inode=99%): [10:44:50] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:45:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:46:18] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:26] / on thyme is UNKNOWN: NRPE: Unable to read output [10:46:26] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:46:27] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:46:39] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [10:46:50] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [10:46:51] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:47:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:05] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:46] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:46] APT on z-dat-s2-b is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:47] /sql on z-dat-s1-b is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:48] RAID on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:48:50] Environment IPMI on thyme is CRITICAL: (Service Check Timed Out) [10:48:51] / on thyme is CRITICAL: (Service Check Timed Out) [10:48:53] Sun Grid Engine execd on wolfsbane is CRITICAL: (Service Check Timed Out) [10:51:05] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [10:51:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 655072.000000 [10:51:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2543030.000000 [10:51:17] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1462048.000000 [10:51:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1677813.000000 [10:51:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [10:51:19] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60444.000000 [10:51:19] SSH on mayapple is CRITICAL: Server answer: [10:51:19] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:51:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:35] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:53] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:56:06] NFS server ha-nfs.esi not responding still trying [11:01:48] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 229134.000000 [11:01:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 262673 MB (4% inode=64%): [11:01:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:01:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:01:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:04] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:02:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:04] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:02:14] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:02:14] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:03] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:03] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:04] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:12:04] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:07] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:07] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:07] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:12:10] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:10] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:10] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:11] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:12:11] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:12:13] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:13] /var on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:13] Environment IPMI on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:13] /var/tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:13] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:13] Environment IPMI on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:14] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:14] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:14] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:28] Sensors on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:28] / on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:35] /tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:35] SRaid on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:35] /var on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:35] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:35] / on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:56] /var/tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:56] Sensors on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:12:56] /tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:20:38] the toolserver is down^^ [11:36:53] I'm getting 504 errors on a few tools [11:36:57] known problem? [11:37:09] ah yes, 'we are down' in the subject :) [11:56:36] Are we down again? [12:09:19] topic [12:10:33] topic is old, status page says we should be up, no hint on the mailing list. hmm. [14:30:10] [[Special:Log/newusers]] create 10 * Лика * (New user account) [14:34:06] Hello all [15:48:19] DaBPunkt, Merlissimo: Hi! The SGE backlog isn't shrinking. I noticed that both nightshade and yarrow are at load 2, even though they have 8 and 4 cores respectively. Can we increase the SGE aim for nightshade to 6 and for yarrow to 3? [15:49:50] that's strange: my ssh-console freezes if I use "top" on any ts-host… [16:15:12] SGE seems to be down, and it wasn't me :-). [16:15:47] No, it's LDAP, it seems. [16:19:32] willow is unavailable again. [16:21:23] now back, but DB replication stopped. *sigh* [16:22:06] krd: I think it has been stopped since yesterday. [16:22:28] i've seen it running for the last hours. [16:22:58] but something seems to be severely broken again. [16:23:11] DaBPunkt: ? [16:24:58] Tried to SSH into yarrow and it connected me to willow instead, which didn't let me log in [16:25:23] SELECT UNIX_TIMESTAMP(MAX(rc_timestamp)) FROM recentchanges; [16:25:24] 1 row in set (2 min 14.72 sec) [16:26:38] krd: Yeah, you're right, on z-dat-s1-b it stopped this morning (054353), on rosemary it has stopped since yesterday evening. [16:37:18] now db seems to be down. let's hope there is some planned action nobody talks about. [16:37:31] which db? [16:37:38] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:37:39] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:37:39] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [16:37:39] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:39] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:49] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:49] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:54] krd@willow:~$ mysql -h dewiki-p.rrdb.toolserver.org [16:37:54] ERROR 1045 (28000): Access denied for user 'krd'@'damiana-bge0.esi.toolserver.org' (using password: NO) [16:37:59] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:59] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:59] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:13] krd@willow:~$ cd [16:38:13] -bash: cd: /home/krd: No such file or directory [16:38:20] the real problem... [16:38:51] I see the problem [16:41:34] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:41:34] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:19:44] hi, is there a problem with SGE? [18:20:27] let me check [18:20:30] qstat is giving an error [18:20:33] Akoopal: Yes. [18:20:36] error: commlib error: can't connect to service (Connection refused) [18:20:39] error: unable to send message to qmaster using port 536 on host "damiana": got send error [18:21:25] DaBPunkt, question, what's database is enwiki_p being replicated from. [18:22:08] how do you mean "what"? What server? [18:22:21] yes. [18:22:45] Cyberpower678: "s1". [18:23:06] I meant on the actual database Wikipedia stores to . [18:23:13] Where is it. [18:23:16] Cyberpower678: The dbxxx? [18:23:29] Not sure what that is? [18:24:09] I'm not sure if I understand you [18:24:20] Wikimedia has (dozens?) of database servers that are named db1, db2, db3, etc. [18:24:25] maybe you should tell us what the probelm is or what you try to do [18:25:21] scfc_de, I want to look up something. It's not really important but it would be nice to know how to connect to the live en.wiki database. [18:25:36] scfc_de: the high numbers are caused by the fact that you do not re-use a server-name for normal. So there is no db001 at the moemnt for example [18:26:09] DaBPunkt: Was just meant as an example. The whole list is available at https://icinga.wikimedia.org/icinga/. [18:26:11] Cyberpower678: https://wiki.toolserver.org/view/Database_access [18:26:21] Cyberpower678: You can't connect to the Wikimedia database servers directly. [18:26:58] DaBPunkt: is cron working ? because I've been keeping receiving cron error mails since 1 or 2 days :/ [18:27:00] * Betacommand trouts Cyberpower678 [18:27:25] Betacommand, what was the trout for? [18:27:41] starting with "error: JSV stderr: Traceback (most recent call last)" [18:28:54] Cyberpower678: why did you think you could connect with the live databases? [18:29:06] Toto_Azero: Those are artifacts of NFS or SGE not working. They go away when the underlying issues are fixed. [18:29:07] Because toolserver can. [18:29:41] scfc_de: ok thanks [18:29:48] Cyberpower678: they are using replication process which isnt the same thing [18:30:03] meh [18:30:09] also its always been a one way process [18:30:19] meh [18:30:59] Cyberpower678: is there something you need to look up for which the API doesn't work? [18:31:12] @replatg [18:31:15] @replag [18:31:23] Nettrom, it wasn't important. [18:32:29] Cyberpower678: ok, no problem... much of what the TS database allows access to can fairly efficiently be retrieved through the WP API [18:41:29] Coren, addshore: New crontab successfully installed. [18:41:52] Cyberpower678: Wrong channel, I think, but "yeay" [18:41:54] Wrong channle [18:59:19] looks like I fixed sge for first. I have now a talk with Silke – so please do not break important things while I'm away ;-) [19:00:24] @notify Merlissimo [19:00:24] I doubt that anyone could have such a nick 'Merlissimo ' [19:00:32] @notify Merlissimo [19:00:32] This user is now online in #wikimedia-tech so I will let you know when they show some activity (talk etc) [21:52:35] DB replication seems to have stopped at 20130513161455 on z-dat-s1-b. [22:38:41] And down we go again... [22:43:19] I hate that… [22:47:06] We all hate that :)