[01:29:29] @replag [01:30:38] * russblau isn't surprised [01:42:39] z-dat-s1-b has caught, though. [04:30:15] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=8008&oldid=7997&rcid=21913 * 185.15.59.202 * (+0) (updated page) [09:39:42] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:39:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:39:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [09:39:43] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:43] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:52] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:53] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:53] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:40:02] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:40:03] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:40:03] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:42:43] MySQL slave on z-dat-s2-b is OK: Uptime: 94240 Threads: 10 Questions: 62199864 Slow queries: 2364 Opens: 686581 Flush tables: 1 Open tables: 256 Queries per second avg: 660.15 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1243 [09:44:13] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:44:13] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:44:13] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:44:13] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:44:13] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:44:43] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [09:44:52] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:44:52] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:45:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:22] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:33] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 544015.000000 [09:45:43] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 694796.000000 [09:45:53] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:46:13] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:48:26] And again... [09:48:42] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2162.000000 [09:49:23] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:02] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:12] /tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:13] SRaid on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:14] Sensors on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:14] / on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:14] /var/tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:15] /var/tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:16] / on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:16] Sensors on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:17] Environment IPMI on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:17] Environment IPMI on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:17] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:18] /var on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:18] /tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:18] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:19] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:22] /var on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:56:26] Sun Grid Engine execd on yarrow is UNKNOWN: Cannot execute /sge/GE/bin/linux-x64/qhost [09:56:26] /var on nightshade is OK: DISK OK - free space: /var 9867 MB (73% inode=48%): [09:56:43] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge/GE/bin/sol-amd64/qstat [09:56:43] /var on yarrow is OK: DISK OK - free space: /var 11664 MB (87% inode=96%): [09:56:43] Environment IPMI on nightshade is OK: ok: temperature ok fan ok voltage ok chassis ok [09:56:43] Load avg. on yarrow is WARNING: WARNING - load average: 13.04, 17.67, 12.80 [09:56:43] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge/GE/bin/sol-amd64/qstat [09:56:44] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge/GE/bin/sol-amd64/qstat [09:56:53] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:56:53] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:56:53] /var/tmp on nightshade is OK: DISK OK - free space: /var/tmp 872 MB (98% inode=99%): [09:56:53] / on nightshade is OK: DISK OK - free space: / 1593 MB (89% inode=94%): [09:56:53] / on yarrow is OK: DISK OK - free space: / 1582 MB (88% inode=94%): [09:56:54] Sensors on yarrow is OK: sensor ok [09:56:54] /tmp on yarrow is OK: DISK OK - free space: /tmp 4086 MB (96% inode=99%): [09:56:55] /var/tmp on yarrow is OK: DISK OK - free space: /var/tmp 827 MB (97% inode=99%): [09:56:55] SRaid on yarrow is OK: OK md0 status=[UU]. [09:56:56] Environment IPMI on yarrow is OK: ok: temperature ok fan ok voltage ok chassis ok [09:56:56] Sensors on nightshade is OK: sensor ok [09:56:57] /tmp on nightshade is OK: DISK OK - free space: /tmp 3343 MB (75% inode=99%): [09:56:57] Sun Grid Engine execd on nightshade is UNKNOWN: Cannot execute /sge/GE/bin/linux-x64/qhost [09:56:58] toolserver.org HTTP on wolfsbane is WARNING: HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.019 second response time [09:57:43] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:57:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:57:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:57:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:57:53] /home on hemlock is OK: DISK OK - free space: /home 13354 MB (26% inode=81%): [09:57:53] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.008 second response time [09:58:13] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [09:58:43] Load avg. on yarrow is OK: OK - load average: 4.24, 13.10, 11.71 [10:01:43] Load avg. on nightshade is WARNING: WARNING - load average: 3.50, 17.02, 20.00 [10:05:53] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2169.000000 [10:06:53] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2177.000000 [10:07:43] Load avg. on nightshade is OK: OK - load average: 3.13, 7.32, 14.59 [10:09:43] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1683.000000 [10:17:12] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [10:29:52] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [10:30:53] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3617.000000 [10:32:53] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3524.000000 [10:32:53] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3520.000000 [10:37:43] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2872 [10:37:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2275 [10:37:53] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3566 [10:38:23] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2141 [10:38:23] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2776 [10:38:43] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [10:38:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [10:38:52] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:38:53] aliasd on yarrow is CRITICAL: Connection refused [10:38:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 166961.000000 [10:39:03] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2712703.000000 [10:39:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1765252.000000 [10:39:13] / on thyme is UNKNOWN: NRPE: Unable to read output [10:39:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:39:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1539862.000000 [10:39:13] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:14] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:39:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:23] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:39:23] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:23] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:39:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:39:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 166953 [10:39:33] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2382 [10:39:33] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:39:33] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 248248 MB (4% inode=63%): [10:39:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:39:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:39:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [10:39:43] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:39:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [10:39:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 880365.000000 [10:39:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:53] NTP on turnera is OK: NTP OK: Offset -0.003631 secs [10:39:53] Sun Grid Engine execd on nightshade is CRITICAL: CRITICAL: execd not communicating [10:39:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:39:53] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:39:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1610011.000000 [10:39:54] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:02] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:02] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:02] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:02] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:03] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:03] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:03] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:12] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [10:40:13] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:22] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:40:23] SSH on mayapple is CRITICAL: Server answer: [10:40:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:40:53] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [10:40:53] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1704.000000 [10:41:13] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [10:41:22] MySQL slave on daphne is OK: Uptime: 2300088 Threads: 55 Questions: 264161271 Slow queries: 484281 Opens: 85327 Flush tables: 1 Open tables: 1927 Queries per second avg: 114.848 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1604 [10:41:32] MySQL slave on z-dat-s6-a is OK: Uptime: 834642 Threads: 7 Questions: 708684215 Slow queries: 28651 Opens: 1236623 Flush tables: 1 Open tables: 3219 Queries per second avg: 849.87 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1772 [10:43:42] MySQL slave on z-dat-s3-a is OK: Uptime: 834603 Threads: 26 Questions: 1000081700 Slow queries: 24509 Opens: 6074070 Flush tables: 1 Open tables: 16384 Queries per second avg: 1198.272 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1776 [10:44:53] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:45:02] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:45:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 695449.000000 [10:45:53] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:46:23] MySQL slave on z-dat-s5-b is OK: Uptime: 98224 Threads: 8 Questions: 280980684 Slow queries: 1167 Opens: 89598 Flush tables: 1 Open tables: 256 Queries per second avg: 2860.611 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1724 [10:46:52] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1600.000000 [10:56:53] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:57:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:57:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:57:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:57:43] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:01:53] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3651 [11:06:43] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3636 [11:25:53] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3599 [11:26:42] Load avg. on z-dat-s2-b is WARNING: WARNING - load average: 17.55, 16.05, 14.01 [11:35:53] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3626 [11:37:43] Load avg. on z-dat-s2-b is OK: OK - load average: 13.82, 14.91, 14.54 [11:38:43] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [11:38:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [11:38:52] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:38:52] aliasd on yarrow is CRITICAL: Connection refused [11:38:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 167648.000000 [11:39:02] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2716302.000000 [11:39:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:02] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1768247.000000 [11:39:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:39:13] / on thyme is UNKNOWN: NRPE: Unable to read output [11:39:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1543306.000000 [11:39:13] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:13] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:39:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:23] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:23] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:23] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:39:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 167675 [11:39:33] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:39:33] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 248099 MB (4% inode=63%): [11:39:43] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:39:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:39:43] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [11:39:43] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [11:39:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 880390.000000 [11:39:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:53] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:39:53] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1613610.000000 [11:39:53] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:40:03] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:40:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:03] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:40:03] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:40:03] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:04] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:40:04] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:40:13] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:40:23] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:40:23] SSH on mayapple is CRITICAL: Server answer: [11:40:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:44:53] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:45:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:45:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:45:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 695206.000000 [11:45:52] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:45:53] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:53] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3593 [14:24:15] hello all [14:30:51] Morning, DaBPunkt [14:31:05] (Well, Morning for me still) [14:31:15] DaBPunkt: Did you get my email re Solaris expertise? [16:01:58] @replag [16:15:34] hi Betacommand [16:15:39] did you get my memo? [16:29:13] closedmouth: the bots should be up [16:29:27] BCBot4 is the exception [16:29:37] its still hosted on the TS [16:29:45] BetacommandBot8 is still missing [16:30:01] closedmouth: what channel? [16:30:08] ##until_it_sleeps-bots [16:32:49] closedmouth: should be up now [16:33:46] nope [16:37:56] closedmouth: for some reason its not joining freenode [16:40:08] closedmouth: its up [16:40:36] thanks [17:30:23] Why are there no email reminders about disk quota? [17:30:29] apparently bots have been down for 3 days [17:32:33] Krinkle: https://jira.toolserver.org/browse/TS-618 [17:36:00] Krinkle: TS is FUBAR [17:36:26] Tell me something I don't know. [17:54:34] Krinkle: because in lowest order it's your own responsibility [17:55:43] valhallasw: In my case the limit was hit due to creation of core dumps. I'm on ~ 400M, and 3 months later of little to no significant changes it grew to 2.2GB, exceeding both the 1GB quota and the 2GB hard limit. [17:56:03] I ran `find` and `rm`, and now Im back to 440M [18:06:45] multichill: I think the technical term is EMFUTN (even more...than normal) [18:14:17] the acronym you're looking for is SNAFU [18:51:17] @replag [18:51:17] Silke_WMDE: s1-rr-a: 11s [-]; s1-rr-a-wd: 2w 22h 21m 15s [+1.00 s/s]; s1-user: 1d 23h 18m 11s [-0.04 s/s]; s1-user-c: 2w 4d 23h 24m 59s [+1.00 s/s]; s1-user-wd: 2w 6d 18h 22m 51s [+1.00 s/s]; s2-user-c: error; s2-user-wd: 2w 4d 3h 53m 7s [+1.00 s/s]; s3-user-wd: 1w 3d 13h 8m 37s [+1.00 s/s] [18:51:18] Silke_WMDE: s4-user-wd: 4w 3d 17h 44m 6s [+1.00 s/s]; s5-rr-a-wd: 6d 14h 14m 26s [+1.00 s/s]; s5-user-c: 6h 56m 30s [+1.00 s/s]; s6-user-wd: 1w 3d 11h 28m 20s [+1.00 s/s]; s7-user-wd: 1w 1d 8h 2m 21s [+1.00 s/s] [19:19:38] Free Memory on turnera is OK: OK - 86.3% (7232580 kB) free. [19:31:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:31:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:31:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:55:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28826.000000 [19:57:17] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:17] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:18] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:38] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:38] / on thyme is UNKNOWN: NRPE: Unable to read output [19:57:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 170054.000000 [19:57:38] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [19:57:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2746222.000000 [19:57:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:57:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1798149.000000 [19:57:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 724109.000000 [19:57:48] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:57:48] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:57:49] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:49] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [19:57:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [19:57:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:57:58] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:58:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:58:18] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:18] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:18] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:18] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:18] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:19] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:19] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [19:58:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1573208.000000 [19:58:38] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1643536.000000 [19:58:38] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:58:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:58:48] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 246311 MB (4% inode=63%): [19:58:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:58:48] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:58:58] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:58] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:58] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:58] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 909345.000000 [19:59:07] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:59:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 170082 [19:59:27] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:59:38] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [19:59:38] SSH on mayapple is CRITICAL: Server answer: [19:59:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:59:58] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:37] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:11:38] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:31:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:31:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:31:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:55:21] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32426.000000 [20:57:18] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:57:19] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:57:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:39] / on thyme is UNKNOWN: NRPE: Unable to read output [20:57:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 171839.000000 [20:57:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [20:57:39] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2749822.000000 [20:57:39] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:57:40] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1801748.000000 [20:57:49] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 727710.000000 [20:57:49] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:49] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:57:49] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:57:49] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [20:57:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [20:57:50] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:58:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:58:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:58:19] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:58:19] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:58:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:19] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:58:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:20] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [20:58:20] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:58:21] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:58:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1576809.000000 [20:58:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:39] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1647136.000000 [20:58:39] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:58:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:58:49] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:58:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 246256 MB (4% inode=63%): [20:58:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:58:49] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:58:58] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:58] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:59:08] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 912949.000000 [20:59:09] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:59:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:59:39] SSH on mayapple is CRITICAL: Server answer: [20:59:39] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:59:39] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:59:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:59:49] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:59:59] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:11:39] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:11:39] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:31:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:31:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:31:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:33:38] Free Memory on damiana is WARNING: WARNING - 5.5% (461428 kB) free! [21:35:39] Free Memory on damiana is OK: OK - 7.6% (638636 kB) free. [21:39:08] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset 11.505836 secs [21:39:39] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45538 MB (7% inode=99%): [21:43:37] Free Memory on damiana is WARNING: WARNING - 5.7% (474616 kB) free! [21:50:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35644.000000 [21:50:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 580387.000000 [21:56:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36086.000000 [21:57:27] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:37] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:57:37] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:57:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:57:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2753429.000000 [21:57:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1805355.000000 [21:57:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:08] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [21:58:08] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731329.000000 [21:58:08] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:58:08] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:58:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:58:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:58:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [21:58:16] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:58:36] / on thyme is UNKNOWN: NRPE: Unable to read output [21:58:37] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 175503.000000 [21:58:38] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [21:58:38] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:58:38] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:58:38] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:58:38] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:58:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:58:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:58:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1650744.000000 [21:58:47] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:57] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:07] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 246203 MB (4% inode=63%): [21:59:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:59:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:59:17] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:17] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [21:59:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1580472.000000 [21:59:46] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:59:47] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:59:56] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:00:07] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 916613.000000 [22:00:08] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:00:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:00:16] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:00:37] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:00:38] SSH on mayapple is CRITICAL: Server answer: [22:00:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:00:38] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:11:56] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:12:37] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:26:32] closedmouth: I saw your mail. I will speak with nosy about it [22:26:35] nacht ts [22:31:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:32:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:32:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:50:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39245.000000 [22:50:10] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 583987.000000 [22:56:40] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39700.000000 [22:57:32] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:42] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:43] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:58:03] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2757042.000000 [22:58:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1808967.000000 [22:58:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [22:58:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:58:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 179103.000000 [22:58:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:46] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:46] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:47] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:48] NTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:56] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:59:06] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [22:59:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1654355.000000 [22:59:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 734989.000000 [22:59:13] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:59:13] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:59:13] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:59:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:59:13] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:59:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 246136 MB (4% inode=63%): [22:59:14] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:59:36] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1584076.000000 [22:59:37] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [22:59:37] / on thyme is UNKNOWN: NRPE: Unable to read output [22:59:37] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [23:00:15] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:00:30] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:00:31] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:31] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 920219.000000 [23:00:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:32] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:32] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:00:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:00:48] SSH on mayapple is CRITICAL: Server answer: [23:01:16] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:01:17] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:01:19] MySQL slave on rosemary is CRITICAL: Lost connection to MySQL server at reading authorization packet, system error: 0 [23:01:21] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:02:57] Load avg. on thyme is CRITICAL: (Service Check Timed Out) [23:03:11] /tmp on thyme is CRITICAL: (Service Check Timed Out) [23:03:13] RAID on thyme is CRITICAL: (Service Check Timed Out) [23:03:14] /mnt on thyme is CRITICAL: (Service Check Timed Out) [23:03:14] / on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:03:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:03:15] APT on yucca is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:48] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40659.000000 [23:13:48] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:13:48] APT on z-dat-s2-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:13:48] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:48] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 246126 MB (4% inode=63%): [23:13:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:13:52] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:13:52] PING on adenia is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [23:13:52] APT on z-dat-s1-b is WARNING: APT WARNING: 35 packages available for upgrade (0 critical updates). [23:13:52] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:01] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:02] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:02] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:02] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:11] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:11] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:11] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:15:22] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:15:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:15:32] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge/GE/bin/sol-amd64/qstat [23:15:51] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge/GE/bin/sol-amd64/qstat [23:15:51] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge/GE/bin/sol-amd64/qstat [23:18:42] /home on hemlock is CRITICAL: DISK CRITICAL - /home is not accessible: No such file or directory [23:19:02] Sun Grid Engine execd on nightshade is UNKNOWN: Cannot execute /sge/GE/bin/linux-x64/qhost [23:19:02] toolserver.org HTTP on wolfsbane is WARNING: HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.012 second response time [23:19:21] Sun Grid Engine execd on yarrow is UNKNOWN: Cannot execute /sge/GE/bin/linux-x64/qhost [23:19:21] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.012 second response time [23:21:11] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:21:21] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:21:22] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:21:32] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:21:32] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:21:42] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:21:42] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:21:52] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:22:02] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:20] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:20] Environment IPMI on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:28] / on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:28] Sensors on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:28] /tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:28] SRaid on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:37] /var on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:37] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:37] / on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:37] /var/tmp on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:37] /tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:38] Sensors on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:52] /var on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:59] Environment IPMI on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:59] /var/tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:59] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:28:19] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:32:59] NFS on ha-nfs.esi is CRITICAL: Connection refused [23:35:53] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:39:01] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2161.000000 [23:41:09] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2190.000000 [23:41:09] Environment IPMI on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:41:28] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:41:59] SSH on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:42:39] And we're down again? [23:43:00] / on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:43:08] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:45:07] Back again. [23:45:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1657093.000000 [23:45:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2759840.000000 [23:45:33] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1811766.000000 [23:45:33] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:34] / on thyme is UNKNOWN: NRPE: Unable to read output [23:45:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:45:35] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset 10.358761 secs [23:45:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1586783.000000 [23:45:36] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:45:36] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:45:53] PING on asw-oe10-esams.mgmt is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [23:45:53] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: No response from remote host fsw2-n1-oe16-esams.mgmt during discovery. [23:45:53] PING on hemlock.mgmt is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [23:45:53] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:54] ethernet 0/1/12 [csw1-esams:1/24] on asw-oe10-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe10-esams.mgmt. [23:46:12] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:12] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:12] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:47:31] just the sge-master and I can go to bed… [23:50:43] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2865.000000 [23:50:43] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2768.000000 [23:50:43] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 2.19, 19.54, 29.39 [23:51:43] Load avg. on yarrow is WARNING: WARNING - load average: 0.19, 9.83, 18.95 [23:53:19] ok, works again. And now finally my bed [23:55:44] Load avg. on yarrow is OK: OK - load average: 0.09, 4.46, 14.65 [23:57:42] Load avg. on nightshade is WARNING: WARNING - load average: 2.25, 6.50, 19.51