[00:00:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[00:00:58] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[00:01:06] <DaBPunkt>	 I have to go to bed now. Nosy will fix the remaining things in the morning
[00:01:10] <DaBPunkt>	 gn8 ts
[00:01:59] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5157.000000  
[00:02:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[00:02:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 578126.000000  
[00:02:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345105 MB (6% inode=63%):  
[00:02:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506859.000000  
[00:02:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[00:02:48] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[00:03:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[00:03:19] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75650 MB (12% inode=99%):  
[00:03:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 361920.000000  
[00:03:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[00:03:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[00:03:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[00:03:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[00:03:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[00:03:49] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[00:03:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[00:04:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[00:04:58] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3071.000000  
[00:05:24] <Alchimista>	 http down again :S
[00:06:58] <tsnag>	 s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 901.000000  
[00:07:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[00:07:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[00:07:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[00:07:58] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[00:08:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[00:08:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[00:08:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[00:08:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[00:08:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:08:58] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[00:09:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:09:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:09:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[00:09:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:09:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:09:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[00:09:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[00:09:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:09:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:09:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:10:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:10:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:10:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[00:13:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[00:15:36] <Jasper_Deng>	 Susan: for what reason was IPv6 disabled on TS?
[00:15:42] <Jasper_Deng>	 it was enabled in 2011
[00:23:48] <apmon>	 Jasper_Deng: https://jira.toolserver.org/browse/TS-1195
[00:25:48] <tsnag>	 s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3311.000000  
[00:27:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[00:28:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512483.000000  
[00:28:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[00:30:05] <Susan>	 NO idea.
[00:30:07] <Susan>	 No idea.
[00:30:48] <tsnag>	 s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1506.000000  
[00:31:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[00:33:04] <Betacommand>	 Susan: hes gone
[00:36:56] <Susan>	 And yet I still don't know. ;-)
[00:38:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[00:49:19] <tsnag>	 MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2172  
[00:57:18] <tsnag>	 MySQL slave on z-dat-s1-b is OK: Uptime: 10768  Threads: 14  Questions: 13440383  Slow queries: 113  Opens: 1846  Flush tables: 1  Open tables: 525  Queries per second avg: 1248.178 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1742  
[01:00:59] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[01:01:00] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:02:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:02:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 577374.000000  
[01:02:49] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 505058.000000  
[01:02:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:02:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[01:03:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[01:03:19] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75279 MB (12% inode=99%):  
[01:03:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 360469.000000  
[01:03:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[01:03:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[01:03:29] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345871 MB (6% inode=63%):  
[01:04:27] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[01:04:39] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[01:04:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[01:04:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[01:04:48] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[01:04:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[01:07:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[01:07:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[01:07:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[01:07:59] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:08:10] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:10] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:10] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:10] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:10] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:10] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:11] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[01:08:49] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[01:08:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[01:08:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:50] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[01:08:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:08:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:09:00] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:09:00] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[01:09:10] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:09:10] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:09:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:09:19] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:09:19] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:09:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[01:09:39] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[01:09:40] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:09:40] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:09:40] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:10:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:10:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:10:30] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[01:13:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[01:27:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[01:28:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512373.000000  
[01:29:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[01:31:29] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[01:38:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[02:01:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[02:01:58] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[02:02:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:02:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 576793.000000  
[02:02:49] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 504268.000000  
[02:02:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:02:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[02:03:10] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[02:03:19] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73080 MB (11% inode=99%):  
[02:03:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 357645.000000  
[02:03:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[02:03:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[02:03:29] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345826 MB (6% inode=63%):  
[02:04:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[02:04:39] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[02:04:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[02:04:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[02:04:48] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[02:05:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[02:07:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[02:07:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[02:08:09] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:08:10] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:08:10] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:08:10] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:08:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:08:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:08:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[02:08:58] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[02:09:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:37] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[02:09:49] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[02:09:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[02:09:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[02:09:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:59] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:09:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:10:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:10:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:10:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[02:10:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:10:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:10:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:10:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[02:10:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[02:10:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:10:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:10:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:11:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:11:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[02:13:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[02:27:29] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[02:28:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 510442.000000  
[02:30:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[02:39:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[03:01:19] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[03:01:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[03:01:59] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:02:49] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 505530.000000  
[03:02:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[03:02:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[03:03:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[03:03:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 577523.000000  
[03:03:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74598 MB (12% inode=99%):  
[03:03:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 352881.000000  
[03:03:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[03:03:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[03:03:29] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345222 MB (6% inode=63%):  
[03:04:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[03:04:29] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[03:04:39] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[03:04:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[03:04:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[03:04:48] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[03:06:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[03:08:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[03:08:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[03:09:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:09] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:09] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[03:09:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[03:09:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[03:09:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[03:09:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:59] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:09:59] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[03:10:00] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:10:00] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[03:10:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:09] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:10:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[03:10:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[03:10:39] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:40] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:10:40] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:11:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:11:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[03:13:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[03:28:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 509172.000000  
[03:28:30] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[03:30:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[03:39:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[04:01:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[04:01:59] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[04:01:59] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[04:03:09] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[04:03:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 578548.000000  
[04:03:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74295 MB (12% inode=99%):  
[04:03:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 345711.000000  
[04:03:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[04:03:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[04:03:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506823.000000  
[04:03:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[04:03:48] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[04:04:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345778 MB (6% inode=63%):  
[04:04:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[04:04:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[04:04:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[04:04:48] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[04:05:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[04:05:29] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[04:06:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[04:08:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[04:08:29] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[04:09:09] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:09:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:09:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:09:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:09:09] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:09:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[04:10:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[04:10:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[04:10:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[04:10:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:59] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:10:59] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[04:11:00] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[04:11:00] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[04:11:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:11:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:11:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:11:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:11:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[04:11:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:11:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:11:37] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[04:11:37] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[04:11:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:11:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:11:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:12:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:12:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[04:14:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[04:29:08] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508660.000000  
[04:29:27] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[04:30:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[04:40:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[05:01:59] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[05:01:59] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[05:03:19] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74118 MB (12% inode=99%):  
[05:03:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 341808.000000  
[05:03:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[05:03:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[05:03:49] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506969.000000  
[05:03:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[05:03:50] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[05:04:09] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[05:04:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 579648.000000  
[05:04:29] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345738 MB (6% inode=63%):  
[05:04:40] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[05:04:40] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[05:04:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[05:04:48] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[05:05:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[05:05:29] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[05:07:00] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[05:08:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[05:08:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[05:09:09] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:09:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:09:10] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:09:10] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:09:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:09:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:10:30] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:10:40] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[05:10:49] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[05:10:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:10:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:10:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[05:10:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:10:50] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:00] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[05:11:00] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:00] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:00] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:00] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[05:11:01] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[05:11:01] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[05:11:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:09] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[05:11:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[05:11:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:11:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:12:17] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[05:12:19] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:12:19] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:12:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:12:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[05:12:50] <jimmyxu>	 @replag
[05:12:50] <tsbot>	 jimmyxu: s1-rr-a: 3m 43s [-0.17 s/s]; s1-rr-a-wd: 5d 4h 8m 42s [+0.38 s/s]; s1-user: 5m 39s [-]; s1-user-wd: 6d 17h 3m 13s [+0.08 s/s]; s2-user-c: error; s2-user-wd: 5d 21h 8m 9s [-0.13 s/s]; s3-user: 3m 28s [+0.01 s/s]; s3-user-wd: 5d 12h 58m 26s [+0.07 s/s]
[05:12:51] <tsbot>	 jimmyxu: s4-user-wd: 5d 20h 49m 15s [-0.00 s/s]; s5-rr-a-wd: 3d 22h 48m 30s [-1.05 s/s]; s6-user-wd: 5d 13h 2m 39s [+0.09 s/s]; s7-user-wd: 5d 19h 10m 5s [-0.35 s/s]
[05:14:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[05:29:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 507931.000000  
[05:29:29] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[05:31:00] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[05:31:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[05:40:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[06:02:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[06:02:58] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[06:03:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[06:04:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 339818.000000  
[06:04:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73893 MB (12% inode=99%):  
[06:04:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[06:04:42] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[06:04:42] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[06:04:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506979.000000  
[06:04:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[06:04:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[06:04:48] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[06:04:48] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[06:05:07] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[06:05:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 580617.000000  
[06:05:30] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345654 MB (6% inode=63%):  
[06:05:30] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[06:06:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[06:07:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[06:08:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[06:08:29] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[06:09:09] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:10] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:10] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:10] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:49] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[06:10:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[06:10:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:10:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:11:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:11:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:11:09] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:11:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:11:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[06:11:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[06:11:39] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:11:39] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:11:39] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:11:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[06:11:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[06:11:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:12:00] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:12:00] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:12:00] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[06:12:00] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[06:12:00] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[06:12:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:12:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[06:12:19] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:12:19] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:12:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:12:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[06:14:30] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[06:30:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508342.000000  
[06:30:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[06:31:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[06:36:37] <sDrewth>	 is it a known issue that the main page for toolserver is blank?
[06:40:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[07:01:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[07:02:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[07:02:58] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[07:04:22] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73748 MB (12% inode=99%):  
[07:04:22] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 340421.000000  
[07:04:22] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[07:04:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[07:04:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508821.000000  
[07:04:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:04:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[07:04:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[07:04:50] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[07:05:09] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:05:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 582716.000000  
[07:05:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[07:05:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[07:05:43] <Susan>	 Lovely...
[07:06:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[07:06:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345485 MB (6% inode=63%):  
[07:06:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[07:07:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[07:09:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[07:09:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[07:10:04] <reba>	 Marlen Caemmerer * Re: [Toolserver-l] Failed Maintenance - was - Re: IP Renumbering of the complete Toolserver cluster
[07:10:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:09] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:09] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[07:11:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[07:11:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:11:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[07:11:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:12:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:12:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[07:12:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[07:12:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:12:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:12:39] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:12:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[07:12:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:12:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:12:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[07:12:59] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[07:12:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:13:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[07:13:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:13:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:13:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:13:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[07:15:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[07:27:58] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2194.000000  
[07:30:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 511123.000000  
[07:30:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[07:32:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[07:41:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[07:50:48] <tsnag>	 s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2185.000000  
[07:51:59] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3633.000000  
[08:01:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[08:02:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[08:02:58] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[08:04:19] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73606 MB (12% inode=99%):  
[08:04:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 343726.000000  
[08:04:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[08:04:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[08:04:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512421.000000  
[08:04:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[08:04:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[08:04:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[08:04:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[08:05:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[08:05:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 586316.000000  
[08:05:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[08:05:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[08:06:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[08:06:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345403 MB (6% inode=63%):  
[08:06:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[08:07:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[08:09:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[08:09:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[08:10:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:11:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[08:11:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:11:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:11:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[08:11:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[08:12:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[08:12:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:12:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[08:12:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:13:37] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:13:37] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:13:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[08:13:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:13:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:13:59] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[08:13:59] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[08:13:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[08:14:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[08:14:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:14:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:14:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:14:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[08:14:48] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000  
[08:15:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[08:30:30] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[08:31:08] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 514783.000000  
[08:33:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[08:41:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[08:52:58] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7293.000000  
[09:01:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[09:02:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[09:02:58] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:04:08] <tsnag>	 SSH on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[09:04:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73396 MB (12% inode=99%):  
[09:04:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347327.000000  
[09:04:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[09:04:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[09:04:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[09:05:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[09:05:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 589916.000000  
[09:05:39] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[09:05:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[09:05:49] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 516081.000000  
[09:05:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[09:05:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[09:05:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[09:06:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 344799 MB (6% inode=63%):  
[09:07:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[09:07:29] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[09:08:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[09:10:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[09:10:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[09:11:09] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:11:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[09:11:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:11:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:11:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[09:11:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:12:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:12:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:12:39] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[09:12:39] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[09:12:39] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:12:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:12:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[09:12:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[09:13:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:13:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[09:13:58] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:13:58] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[09:14:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:14:19] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:14:19] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:14:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:14:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[09:14:48] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7225.000000  
[09:15:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[09:31:08] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518383.000000  
[09:31:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[09:34:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[09:41:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[09:52:58] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10893.000000  
[10:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[10:03:59] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:04:09] <tsnag>	 SSH on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[10:04:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[10:05:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[10:05:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 593517.000000  
[10:05:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 350986.000000  
[10:05:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73166 MB (12% inode=99%):  
[10:05:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[10:05:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[10:05:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[10:05:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[10:05:49] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 519680.000000  
[10:06:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345085 MB (6% inode=63%):  
[10:06:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[10:06:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[10:06:48] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[10:07:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[10:08:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[10:08:39] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 25190 MB (2% inode=99%):  
[10:09:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[10:10:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[10:10:29] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[10:11:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[10:11:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:11:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:11:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[10:11:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:12:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:12:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:12:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:12:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:12:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[10:13:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[10:13:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[10:13:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[10:13:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:13:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[10:13:58] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:13:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[10:14:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:14:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:14:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:14:47] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10825.000000  
[10:15:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[10:15:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:15:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[10:31:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 521983.000000  
[10:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[10:31:29] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[10:34:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[10:41:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[10:44:58] <tsnag>	 SSH on yucca is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0)  
[10:52:59] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14493.000000  
[11:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[11:03:58] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:05:09] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[11:05:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 597116.000000  
[11:05:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[11:05:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 523280.000000  
[11:05:49] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[11:06:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354647.000000  
[11:06:19] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72960 MB (11% inode=99%):  
[11:06:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[11:06:29] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 343053 MB (6% inode=63%):  
[11:06:39] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[11:06:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[11:06:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[11:06:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[11:06:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[11:08:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[11:08:29] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[11:09:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[11:10:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[11:11:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[11:11:49] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[11:11:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:11:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:11:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[11:11:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:12:00] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 15.89, 15.46, 14.48  
[11:12:09] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:12:09] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:12:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:12:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[11:13:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:39] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[11:13:39] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[11:13:39] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:39] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:39] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:40] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[11:13:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:13:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[11:13:58] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:13:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[11:14:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:14:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:14:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:14:49] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14425.000000  
[11:15:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[11:15:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:15:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[11:20:59] <tsnag>	 Load avg. on daphne is OK: OK - load average: 14.29, 14.89, 14.66  
[11:31:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 525583.000000  
[11:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[11:32:29] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[11:34:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[11:41:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[11:53:58] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18154.000000  
[12:01:58] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 14.65, 15.15, 14.87  
[12:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[12:04:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:04:59] <tsnag>	 Load avg. on daphne is OK: OK - load average: 14.59, 14.98, 14.85  
[12:05:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 526880.000000  
[12:05:58] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[12:06:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[12:06:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 600777.000000  
[12:06:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[12:07:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 358307.000000  
[12:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72753 MB (11% inode=99%):  
[12:07:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[12:07:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 342781 MB (6% inode=63%):  
[12:07:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[12:07:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[12:07:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[12:07:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[12:07:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[12:09:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[12:09:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[12:09:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[12:11:27] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[12:11:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[12:12:09] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:12:09] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[12:12:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:12:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:12:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[12:12:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:12:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:10] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:10] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[12:13:59] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:13:59] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[12:14:00] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:14:00] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[12:14:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:14:19] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:14:19] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:14:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:14:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[12:14:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[12:14:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:14:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:14:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:14:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[12:14:57] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 15.86, 15.48, 15.11  
[12:15:48] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18085.000000  
[12:16:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[12:16:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:16:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[12:16:59] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1974.000000  
[12:16:59] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1974  
[12:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[12:32:08] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 529243.000000  
[12:32:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[12:32:49] <Alchimista>	 @replag
[12:32:50] <tsbot>	 Alchimista: s1-rr-a-wd: 5d 10h 33m 40s [+0.87 s/s]; s1-user: 37m 29s [+0.07 s/s]; s1-user-wd: 6d 23h 19m 44s [+0.86 s/s]; s2-user-c: error; s2-user-wd: 6d 3h 1m 33s [+0.80 s/s]; s3-user: 1m 24s [-0.00 s/s]; s3-user-wd: 5d 16h 48m 3s [+0.52 s/s]; s4-user-wd: 6d 2h 48m 30s [+0.82 s/s]
[12:32:51] <tsbot>	 Alchimista: s5-rr-a-wd: 4d 3h 57m 27s [+0.70 s/s]; s5-user: 2m 12s [+0.00 s/s]; s5-user-c: 5h 18m 35s [-0.06 s/s]; s6-user: 37m 26s [-]; s6-user-wd: 5d 19h 25m 11s [+0.87 s/s]; s7-user-wd: 6d 1h 2m 3s [+0.80 s/s]
[12:34:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[12:38:04] <reba>	 DeltaQuad Wikipedia * Re: [Toolserver-l] Failed Maintenance - was - Re: IP Renumbering of the complete Toolserver cluster
[12:39:08] <tsnag>	 /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 87436 MB (8% inode=99%):  
[12:39:58] <tsnag>	 Load avg. on daphne is OK: OK - load average: 13.76, 14.40, 14.95  
[12:42:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[12:53:59] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21753.000000  
[13:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.
[13:05:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:05:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 530480.000000  
[13:06:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[13:06:58] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[13:07:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:07:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 604436.000000  
[13:07:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 361907.000000  
[13:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72581 MB (11% inode=99%):  
[13:07:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[13:07:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 342359 MB (6% inode=63%):  
[13:07:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[13:07:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[13:07:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:07:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[13:07:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[13:09:54] <Alchimista>	 http problem seems to persist -> https://toolserver.org/~alchimista
[13:09:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[13:10:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[13:10:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[13:11:29] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[13:11:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[13:12:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:12:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[13:12:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:12:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:12:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[13:12:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:12:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:12:59] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 15.73, 15.26, 14.31  
[13:13:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[13:13:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:13:59] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[13:13:59] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:13:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[13:14:08] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:14:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:14:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:14:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[13:14:39] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[13:14:39] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:14:39] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:14:39] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:14:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[13:15:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:15:48] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21685.000000  
[13:16:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[13:16:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:16:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[13:16:58] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2939.000000  
[13:16:59] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2940  
[13:22:58] <tsnag>	 Load avg. on daphne is OK: OK - load average: 13.75, 14.73, 14.61  
[13:26:08] <tsnag>	 /sql on z-dat-s1-b is OK: DISK OK - free space: /sql 112713 MB (11% inode=99%):  
[13:29:09] <tsnag>	 MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1918  
[13:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[13:32:08] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 532832.000000  
[13:32:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[13:34:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[13:37:58] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3614.000000  
[13:37:58] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3614  
[13:43:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[13:53:58] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15275.000000  
[14:03:48] <tsnag>	 s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3498.000000  
[14:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[14:05:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:05:48] <tsnag>	 s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1669.000000  
[14:05:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 533138.000000  
[14:06:58] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[14:07:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[14:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72332 MB (11% inode=99%):  
[14:07:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 362489.000000  
[14:07:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[14:07:27] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[14:07:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[14:07:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[14:07:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[14:07:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[14:07:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[14:08:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 607269.000000  
[14:08:29] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 342223 MB (6% inode=63%):  
[14:09:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[14:10:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[14:11:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[14:12:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[14:12:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[14:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[14:12:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[14:13:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:13:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[14:13:58] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:13:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[14:14:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:17] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:27] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:14:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[14:14:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[14:15:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:15:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[14:15:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:15:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:15:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:15:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[14:17:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[14:17:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:17:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[14:17:58] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3130.000000  
[14:20:58] <tsnag>	 s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1701.000000  
[14:23:08] <tsnag>	 MySQL slave on z-dat-s2-b is OK: Uptime: 61411  Threads: 14  Questions: 76327905  Slow queries: 974  Opens: 539782  Flush tables: 1  Open tables: 256  Queries per second avg: 1242.902 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795  
[14:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[14:32:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 535139.000000  
[14:32:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[14:34:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[14:37:59] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5495.000000  
[14:37:59] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5498  
[14:43:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[15:03:59] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.
[15:05:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[15:05:49] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 534932.000000  
[15:06:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[15:07:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72109 MB (11% inode=99%):  
[15:07:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[15:07:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[15:07:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[15:08:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 609391.000000  
[15:08:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 360097.000000  
[15:08:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[15:08:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:08:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[15:08:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[15:09:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 342073 MB (6% inode=63%):  
[15:09:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[15:11:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[15:11:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[15:12:27] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[15:12:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[15:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[15:12:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:13:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:13:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:13:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:13:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[15:13:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:13:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:13:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:00] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:00] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:00] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[15:14:00] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[15:14:00] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:14:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:10] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:10] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:10] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:20] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:14:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[15:14:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[15:15:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[15:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:15:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[15:15:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:15:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:15:39] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:15:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[15:17:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[15:17:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:17:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[15:21:47] <apmon>	 Is there any news on what is causing the http failures?
[15:28:40] <DarkoNeko>	 dunno, but at least the crons are working properly
[15:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[15:32:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[15:33:08] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 536265.000000  
[15:33:16] <Betacommand>	 apmon: last I heard it was the load balancers
[15:34:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[15:38:57] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7439.000000  
[15:38:58] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7440  
[15:41:58] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 15.32, 15.04, 13.70  
[15:43:58] <tsnag>	 Load avg. on daphne is OK: OK - load average: 14.94, 14.96, 13.83  
[15:44:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[16:02:58] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 15.27, 15.04, 14.54  
[16:03:59] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.
[16:05:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[16:06:48] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 535898.000000  
[16:06:58] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[16:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71931 MB (11% inode=99%):  
[16:07:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[16:07:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[16:07:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[16:08:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[16:08:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 357138.000000  
[16:08:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[16:08:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[16:08:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[16:08:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[16:09:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 611355.000000  
[16:09:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 341857 MB (6% inode=63%):  
[16:09:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[16:11:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[16:11:29] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[16:12:29] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[16:12:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[16:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[16:12:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:13:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:13:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:13:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:13:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:13:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:13:59] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[16:13:59] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[16:13:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[16:14:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:14:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[16:14:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:14:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[16:14:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:14:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:14:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[16:15:08] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:09] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:10] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:15:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[16:15:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:16:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[16:16:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:16:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:16:39] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:16:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[16:18:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[16:18:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:18:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[16:30:58] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 15.46, 15.14, 15.02  
[16:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[16:32:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[16:33:08] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 536765.000000  
[16:33:58] <tsnag>	 Load avg. on daphne is OK: OK - load average: 14.77, 14.95, 14.97  
[16:34:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[16:38:58] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8692.000000  
[16:38:59] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8690  
[16:41:59] <tsnag>	 Load avg. on daphne is WARNING: WARNING - load average: 15.89, 15.73, 15.35  
[16:44:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[17:01:59] <tsnag>	 Load avg. on daphne is OK: OK - load average: 13.61, 14.62, 15.00  
[17:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[17:06:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:06:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[17:06:59] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 537704.000000  
[17:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71681 MB (11% inode=99%):  
[17:07:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[17:07:43] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[17:07:43] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[17:08:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[17:08:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 358880.000000  
[17:08:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[17:09:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 614522.000000  
[17:09:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 341163 MB (6% inode=63%):  
[17:09:47] <tsnag>	 s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2183.000000  
[17:09:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[17:09:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[17:09:48] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[17:09:59] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2196.000000  
[17:10:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[17:11:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[17:11:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[17:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[17:12:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:13:11] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:13:11] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:13:29] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[17:13:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[17:14:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:14:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[17:14:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:14:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[17:14:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:14:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:14:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[17:14:59] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:14:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:00] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:00] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[17:15:00] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:15:00] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[17:15:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:09] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:15:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:16:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:16:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[17:16:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:16:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:16:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:16:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[17:18:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[17:18:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:18:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[17:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[17:32:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[17:33:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 540360.000000  
[17:33:48] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3623.000000  
[17:33:58] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3637.000000  
[17:34:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[17:39:58] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9087.000000  
[17:39:59] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9087  
[17:44:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[18:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.
[18:06:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[18:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71485 MB (11% inode=99%):  
[18:07:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[18:07:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[18:07:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[18:07:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[18:07:59] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 539240.000000  
[18:08:09] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[18:08:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 362480.000000  
[18:08:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[18:09:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 618122.000000  
[18:09:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 341007 MB (6% inode=63%):  
[18:09:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[18:09:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[18:09:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[18:11:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[18:12:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[18:12:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[18:12:47] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[18:12:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:13:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[18:13:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[18:14:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:14:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[18:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[18:15:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[18:15:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[18:15:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:15:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[18:15:59] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[18:15:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[18:16:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:16:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:16:08] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:16:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:16:08] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:16:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:16:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[18:17:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:17:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:17:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:17:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[18:18:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[18:18:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:18:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[18:31:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[18:32:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[18:33:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 543960.000000  
[18:33:48] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7223.000000  
[18:33:58] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7236.000000  
[18:34:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[18:39:59] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10315.000000  
[18:39:59] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10313  
[18:44:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[19:03:48] <tsnag>	 s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3195.000000  
[19:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[19:05:58] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3468.000000  
[19:06:09] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[19:06:48] <tsnag>	 s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1663.000000  
[19:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71178 MB (11% inode=99%):  
[19:07:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[19:07:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[19:07:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[19:08:11] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[19:08:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 364472.000000  
[19:08:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[19:08:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[19:08:59] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 541016.000000  
[19:09:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 621194.000000  
[19:09:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340866 MB (6% inode=63%):  
[19:10:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[19:10:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[19:10:48] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[19:11:58] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[19:11:58] <tsnag>	 s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1687.000000  
[19:12:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[19:12:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[19:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[19:12:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:13:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[19:13:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[19:13:54] <Magog_the_Ogre>	 why am I getting 404s for all the pages i'm trying to load?
[19:14:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:14:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:14:25] <Susan>	 Bad luck?
[19:14:37] <Susan>	 If you tell us what pages you're trying to load, it would help.
[19:14:40] <Susan>	 Magog_the_Ogre: ^
[19:14:54] <Magog_the_Ogre>	 actually, it's intermittent
[19:14:58] <Magog_the_Ogre>	 sometimes I try to click it, and it works
[19:15:01] <Magog_the_Ogre>	 other times it doesn't
[19:15:04] <Magog_the_Ogre>	 maybe one of the nodes is down?
[19:15:05] <Magog_the_Ogre>	 https://toolserver.org/~magog/fileinfo.php
[19:15:07] <Susan>	 It's the Sabbath.
[19:15:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[19:15:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:24] <Magog_the_Ogre>	 (or is based in Israel?!)
[19:15:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:34] <Susan>	 :-)
[19:15:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[19:15:42] <Susan>	 https://toolserver.org/~magog/fileinfo.php consistently loads for me.
[19:15:47] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[19:15:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:57] <Magog_the_Ogre>	 try again in a minute
[19:15:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[19:15:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:15:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[19:16:01] <Magog_the_Ogre>	 and then in another minute if it loads
[19:16:08] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:16:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:16:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:16:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:16:08] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:16:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:16:19] <Susan>	 Hmmm.
[19:16:33] <Susan>	 $ curl https://toolserver.org/~magog/fileinfo.php
[19:16:33] <Susan>	 curl: (35) error:14077458:SSL routines:SSL23_GET_SERVER_HELLO:reason(1112)
[19:16:37] <Susan>	 wtf
[19:16:37] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[19:16:42] <Susan>	 A subsequent curl request worked.
[19:16:48] <Susan>	 Yeah, something is a little funky there. :-)
[19:16:56] <Susan>	 You may be able to use HTTP with more luck.
[19:16:58] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[19:16:58] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[19:16:58] <Susan>	 If it's SSL-related.
[19:17:10] <Susan>	 The Toolserver is kind of falling apart.
[19:17:35] <Magog_the_Ogre>	 I have set all toolserver.org to automatically be SSL in my browser
[19:17:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:17:43] <Susan>	 I forgive you.
[19:18:04] <Susan>	 HTTPSEverywhere is doing that for me as well.
[19:18:06] <Magog_the_Ogre>	 I've used public wifi too much in the past to be pasting my password all over for everyone to see
[19:18:10] <Susan>	 But I'm not trying to use your tool very much.
[19:18:21] <scfc_de>	 I looks like it's SSL-related, as prefixing ortelius. or wolfsbane. and converting it to http work fine.
[19:18:21] <Susan>	 For a file upload history tool?
[19:18:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[19:18:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:18:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:18:32] <Magog_the_Ogre>	 well not that one
[19:18:34] <Magog_the_Ogre>	 other pages use it
[19:18:39] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:18:39] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[19:18:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[19:18:43] <Magog_the_Ogre>	 like this: https://toolserver.org/~magog/oldver.php
[19:18:46] <Susan>	 You can tunnel through the Toolserver if you want, I imagine.
[19:18:48] <Magog_the_Ogre>	 (which incidentally is not loading for me)
[19:18:50] <Susan>	 To not expose your password.
[19:19:00] <Magog_the_Ogre>	 but I can't use the web interface
[19:19:11] <Magog_the_Ogre>	 or commonshelper: https://toolserver.org/~magnus/commonshelper.php
[19:19:16] <Susan>	 You can tunnel all Web traffic with ssh -F.
[19:19:18] <Susan>	 I used to do it.
[19:19:35] <Susan>	 Hmmmm.
[19:19:41] <Susan>	 What happened to CommonsHelper? :-/
[19:19:51] <Magog_the_Ogre>	 you see, it's not my page
[19:19:56] <Susan>	 https://toolserver.org/~mzmcbride/stalker/ :-(
[19:19:58] <Magog_the_Ogre>	 it's the page loader
[19:20:05] <Susan>	 The Toolserver is falling apart.
[19:20:06] <Susan>	 As I said.
[19:20:16] <Susan>	 We must accept that all things that live must eventually die.
[19:20:45] <Susan>	 Yeah, the Web servers seem pretty well fucked.
[19:20:48] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2210.000000  
[19:20:52] <Susan>	 Not sure if it's HTTPS or something else.
[19:21:13] <Susan>	 Seems like it's HTTPS/SSL.
[19:21:17] <Susan>	 Someone should file a ticket in JIRA.
[19:22:26] <Magog_the_Ogre>	 heh
[19:22:37] <Magog_the_Ogre>	 I see you're speaking in the third person, when you rightfully should not be
[19:23:01] <Magog_the_Ogre>	 not sure if that was supposed to be second or first person, susan
[19:23:07] <Susan>	 So many persons.
[19:23:14] <Susan>	 I'm not filing a ticket, I'm going to make lunch. :-)
[19:23:28] <Susan>	 And come to terms with the slow death of the Toolserver.
[19:23:46] <FunPika>	 https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs Just have to keep waiting for this to put it out of it's misery ;)
[19:24:19] <Susan>	 Or bring further misery.
[19:24:20] <Susan>	 We'll see!
[19:25:49] <Magog_the_Ogre>	 so why is the toolserver falling apart?
[19:26:12] <Coren>	 I don't see it as "put toolserver out of its misery" so much as take the relay after a long and distinguished career ending in a well-deserved retirement.  :-)
[19:29:37] <Susan>	 Magog_the_Ogre: Too many users, not enough resources, I think.
[19:29:41] <Susan>	 People + hardware.
[19:29:50] <Susan>	 Plus WMF wants to set up its own walled garden called Labs.
[19:30:04] <Susan>	 And they control the data stream (DB replication).
[19:30:04] <Susan>	 In a nutshell.
[19:30:34] <Magog_the_Ogre>	 my application doesn't bother with that DB nonsense
[19:30:38] <Magog_the_Ogre>	 it's all API
[19:30:59] <Susan>	 Then you'd be about 1,000 times better off getting shared hosting or a VPS.
[19:31:02] <Susan>	 Far more reliable. :-)
[19:31:07] <Susan>	 Nothing in life is free. ;-)
[19:31:15] <Susan>	 The Toolserver is free, but not very stable.
[19:31:19] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[19:31:31] <Susan>	 For a few dollars a month, you could have a lot more stability.
[19:31:45] <Magog_the_Ogre>	 maybe
[19:31:47] <Magog_the_Ogre>	 but you remember
[19:31:51] <Magog_the_Ogre>	 I'm volunteering my time too
[19:31:58] <Magog_the_Ogre>	 it's not like I'm using it for my own purposes
[19:31:58] <Susan>	 Sure.
[19:32:04] <Magog_the_Ogre>	 it's all educational for Wikimedia, etc.
[19:32:08] <Susan>	 I need we need more Toolservers.
[19:32:16] <Susan>	 Err, I think we need. *
[19:32:25] <Susan>	 Maybe I'll set one up.
[19:32:31] <valhallasw>	 Susan, Magog_the_Ogre, there was major maintenance yesterday evening (CEST)
[19:32:32] <scfc_de>	 Magog_the_Ogre: If you only use the API, you can migrate to Tool Labs now.
[19:32:46] <Magog_the_Ogre>	 scfc_de, noted
[19:32:53] <Coren>	 Susan: Where do you get that "not reliable" idea from?
[19:33:02] <Susan>	 Coren: Having used the Toolserver for a few years. :-)
[19:33:08] <Magog_the_Ogre>	 lol
[19:33:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 545168.000000  
[19:33:16] <Susan>	 I'm not placing blame.
[19:33:18] <Susan>	 And I'm not spiteful.
[19:33:20] <Susan>	 I realize it's a free service.
[19:33:21] <scfc_de>	 Susan: If there was a constant public stream of DB replication - that would be cool :-).
[19:33:23] <valhallasw>	 I think the only thing 'not reliable' is people who do not renew their account
[19:33:28] <Susan>	 scfc_de++++
[19:33:31] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[19:33:38] <Susan>	 valhallasw: That's certainly part of it. :-)
[19:33:42] <Susan>	 But it also has corrupt DBs.
[19:33:43] <valhallasw>	 and db replication, of course
[19:33:46] <Susan>	 Because DB replication is annoying.
[19:33:53] <Coren>	 Susan: the ts never had the resources to work hard on reliability.  Litte hardware and no infrastructure.
[19:33:55] <Susan>	 And sometimes the system gets overloaded.
[19:34:00] <valhallasw>	 Yep, but your VPS won't even give you bad DB replication ;-)
[19:34:04] <Susan>	 There isn't enough resource monitoring, so one person can unfairly use too much.
[19:34:07] <Susan>	 Even inadvertently.
[19:34:11] <Coren>	 The sysadmins already did miracles with what little they had
[19:34:17] <Susan>	 valhallasw: Well, as scfc_de says, we need to fix that.
[19:34:30] <Susan>	 Coren: I'm not blaming, as I said. But when you ask why I think it's not reliable...
[19:34:34] <Magog_the_Ogre>	 susan, I read that as corrupt DBAs
[19:34:39] <Susan>	 Heh.
[19:34:41] <Susan>	 Those are bad too. ;-)
[19:34:42] <Magog_the_Ogre>	 I was thinking, the problem is human corruption?
[19:34:47] <Susan>	 No, the people are all great.
[19:34:51] <Susan>	 DaB. is wonderful.
[19:34:54] <Susan>	 River was wonderful.
[19:34:57] <Susan>	 But they're volunteers.
[19:34:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[19:35:03] <Susan>	 Too. 
[19:35:13] <valhallasw>	 Susan: this is not trivial - afaik replication can only replicate entire tables, and there is private info that needs to be filtered out. But it would be great to have, certainly.
[19:35:17] <Susan>	 Maybe Labs will be a unicorn-filled wonderland. ;-)
[19:35:36] <Susan>	 valhallasw: Those are solvable problems, though.
[19:35:45] <Susan>	 I didn't say they were trivial. DB replication is annoying.
[19:35:50] <Susan>	 But they're gonna have to redo it anyway.
[19:35:51] <Susan>	 For Labs.
[19:35:58] <Susan>	 So a scalable architecture would be better.
[19:36:09] <scfc_de>	 valhallasw: There are other replication schemes apart from MySQL; you don't have to pass them around in binlogs.
[19:36:17] <valhallasw>	 scfc_de: true.
[19:36:18] <Susan>	 Let people create their own Toolservers on AWS or wherever.
[19:36:40] <valhallasw>	 Susan: there's not that much that needs to be redone - simple replication is something the WMF is doing *a lot*
[19:36:57] <valhallasw>	 and the VIEW definitions have been thought about by the TS for a while now :-)
[19:37:12] <Coren>	 Susan: My point is the labs is designed and has the resources for reliability. You can't expect maintainers to also be sysadmins.
[19:38:42] <Susan>	 valhallasw: First you said it wasn't trivial. Then you said there wasn't that much to be done. ;-)
[19:38:59] <Susan>	 Coren: My point is that we don't really want to put all of our eggs in one basket.
[19:39:17] <Susan>	 Coren: And the secondary point was about stability and reliability vs. paying for something.
[19:39:22] <valhallasw>	 Susan: it's the *public* replication that's hard
[19:39:29] <valhallasw>	 not the replication per se
[19:39:29] <Susan>	 When it's a paid service, it is different, I promise.
[19:39:41] <valhallasw>	 but I did not formulate it that clearly :-)
[19:39:43] <scfc_de>	 It would be interesting to see if an enhanced "recent changes feed" that people were discussing on wikitech-l would be "enough" to be useful for other "toolservers".  To query the MediaWiki tables is more folklore than really what makes sense.
[19:39:45] <Susan>	 valhallasw: Because of physical server proximity? Or what?
[19:39:59] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11134.000000  
[19:39:59] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11134  
[19:40:11] <Susan>	 scfc_de: XMPP had the promise of something like that, kind of.
[19:40:13] <valhallasw>	 Susan: no, because the way replication works now means everything is copied. On the TS, the users only get a filtered view of that data
[19:40:18] <Coren>	 Susan: You're confusing "the wmf pays for it" with "free" :-)
[19:40:28] <Susan>	 valhallasw: Right, but you can redo that. Labs seems to be redoing that.
[19:40:32] <valhallasw>	 basically, you don't want stuff like hashed passwords and deleted revisions to leak out
[19:40:33] <Susan>	 valhallasw: Have you read what they're doing?
[19:40:47] <Susan>	 Coren: How was the Toolserver supported?
[19:41:02] <valhallasw>	 Susan: yes, but I think that means the database servers will not be volunteer-run, but WMF-run
[19:41:09] <Susan>	 Right...
[19:41:26] <Susan>	 we need to try to move away from that model.
[19:41:33] <Susan>	 Rather than consolidating everything under the WMF.
[19:41:38] <Susan>	 It's bad policy.
[19:42:21] <Coren>	 By volunteers, on resources partially subsidised by WMDE and the WMF.  The labs are integrated under the WMF operations, and get the same level of resources other internal projects have.
[19:44:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[19:45:32] <Susan>	 Heh, other projects like Wikisource and Wikiquote.
[19:45:48] <Susan>	 I'm not going to try to get you to speak ill of your current employer.
[19:45:54] <Susan>	 But I think we can all agree we can do better.
[19:46:19] <Susan>	 Magog_the_Ogre: I'll see about getting another Toolserver up and running.
[19:46:34] <Susan>	 As a distributed cost, basic cheap hosting among Wikimedians would probably be pretty cheap.
[19:46:41] <Susan>	 Not cheap, even.
[19:46:42] <Magog_the_Ogre>	 and I'll see about porting my tools the heck out of here
[19:47:03] <Susan>	 But there are additional costs like SSL to consider.
[19:47:12] <Susan>	 And getting a volunteer sysadmin.
[19:48:06] <Susan>	 I wonder if I could get the local Wikimedia chapter to do it. :-)
[19:49:26] <Susan>	 Hmmmm.
[19:49:33] <Susan>	 Wikimedia DC could set up an American Toolserver.
[19:49:41] <Susan>	 To match the Polish Toolserver and the German Toolserver.
[19:49:49] <Susan>	 Food for thought. :-)
[19:52:28] <Magog_the_Ogre>	 great idea
[19:52:37] <Magog_the_Ogre>	 if I ever go to WM DC I'll bring it up
[19:53:46] <Susan>	 WMDC is meeting this evening.
[20:03:59] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[20:06:09] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[20:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70915 MB (11% inode=99%):  
[20:07:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[20:07:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[20:07:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[20:08:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[20:08:59] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 542326.000000  
[20:09:09] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[20:09:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 622196.000000  
[20:09:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 361412.000000  
[20:09:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[20:09:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340667 MB (6% inode=63%):  
[20:10:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[20:10:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[20:10:48] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[20:11:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[20:12:09] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[20:12:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[20:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[20:12:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:13:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[20:13:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[20:14:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:14:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:18] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[20:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[20:15:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[20:15:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[20:15:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:15:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[20:16:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:16:08] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:16:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:16:08] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:16:08] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:16:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[20:16:57] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[20:16:58] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[20:17:08] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:17:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:19:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[20:19:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:19:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:19:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:19:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[20:19:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[20:23:04] <Susan>	 I put some thoughts here: https://meta.wikimedia.org/wiki/American_Toolserver
[20:33:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 545472.000000  
[20:34:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[20:35:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[20:39:59] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12293.000000  
[20:40:00] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12297  
[20:45:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[20:56:44] <scfc_de>	 Susan: Not my turf, but I find the "Support" bit the wrong way around: Developers and maintainers of tools should pay that Wikipedians can use them for free?  Developers would need a *very* big ego so that that works :-).
[20:57:22] <Susan>	 Hmmmm.
[20:57:40] <Susan>	 That section could probably use work.
[20:57:51] <Susan>	 I think I'd be willing to pay a small fee for an account.
[20:57:56] <Susan>	 But perhaps I'm unusual.
[20:58:05] <scfc_de>	 Probably :-).
[21:01:18] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[21:03:59] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.
[21:06:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[21:07:18] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70707 MB (11% inode=99%):  
[21:07:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[21:07:53] <Coren>	 Susan: You're at least unusual enough in that you feel comfortable to sysadmin your own infrastructure.  I expect few tools maintainers feel the same.
[21:08:24] <Susan>	 Coren: You don't have a VPS?
[21:08:38] <Susan>	 I'm not sure what's so radical about the concept of shared hosting.
[21:08:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[21:08:38] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[21:08:49] <Coren>	 Sure, but I'm a syadmin by trade.  And it's not virtual, it's a co-lo box (but that's besides the point).
[21:09:09] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[21:09:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 623223.000000  
[21:09:14] <Susan>	 I'm not saying volunteers should be setting up the next Google.
[21:09:28] <Coren>	 Susan: There isn't, but I'd be hard pressed to find a justification to pay to have an unmaintained infrastructure when I can get a maintained one for free.  :-)
[21:09:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340476 MB (6% inode=63%):  
[21:09:30] <Susan>	 But the Toolserver seems to have difficulty providing basic Web hosting.
[21:09:42] <Susan>	 I have a VPS I use that seems more stable.
[21:09:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[21:09:59] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 543911.000000  
[21:10:06] <Susan>	 Obviously there aren't as many people using it, of course.
[21:10:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 359046.000000  
[21:10:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[21:10:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[21:10:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[21:10:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[21:10:57] <Coren>	 Susan: You aren't being fair on the TS admins either; there's only 2, and they are both part-time, and the hardware and OS is old and has accrued a significant maintenance debt.
[21:11:09] <Susan>	 I'm not trying to be unfair to anyone.
[21:11:16] <Susan>	 I think you keep reading me as blaming people.
[21:11:30] <Susan>	 But if you remove the database replication and focus just on hosting,
[21:11:34] <Susan>	 I think you could do a lot with a little.
[21:11:45] <Coren>	 Susan: Sure.  Hence:  Tool Labs!
[21:11:51] <Susan>	 The vast majority of Toolserver users use very few resources.
[21:11:59] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[21:12:11] <Susan>	 Coren: Uhhh, again, putting everything under a single WMF umbrella isn't a great idea.
[21:12:28] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[21:12:29] <Coren>	 Susan: Except for general dislike of WMF, why is that?
[21:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[21:13:01] <Susan>	 Is that not enough?
[21:13:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[21:13:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[21:13:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[21:13:48] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:14:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:14:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:17] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[21:15:18] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:18] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:28] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:38] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[21:15:42] <Coren>	 No, it isn't, because it's neither rational nor well-founded; the efforts being deployed to provide a stable home for community maintained tools is large, and the result is strong.  You are rejecting it out of hand without a second glance because you dislike the WMF and not because of any characteristic of the Tool Labs /itself/
[21:15:48] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:48] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[21:15:48] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:48] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:58] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[21:15:58] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:58] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:15:58] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:16:08] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:16:36] <valhallasw>	 Coren: the fact it's the same organisation as the main host is a reason on itself
[21:16:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[21:16:58] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[21:16:59] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[21:16:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[21:17:07] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:17:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:17:08] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:17:08] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:17:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:17:12] <valhallasw>	 say WMF goes bankrupt - how are you going to restore wikipedia?
[21:17:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:17:59] <Coren>	 valhallasw: I don't see it.  In fact, I see the opposite: the closer integration between the labs and the actual source/destination of the data is a net positive, so is the interaction between operations, dev, and community maintainers.
[21:18:49] <Coren>	 valhallasw: That question is answered the same way regardless of where any tools are located:  start a new data center and restore from last dumps.  But it's also ridiculous; even *if* the WMF went boom, the idea that nobody would take the infrastructure over to keep it running is laughable.
[21:19:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[21:19:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:19:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:19:38] <Coren>	 Anyone who has the resources to restore wikipedia could do it cheaper and simpler by simply taking over the same infrastructure.  The equipment and data centers wouldn't just vanish in a puff of smoke.
[21:19:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:19:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[21:19:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[21:21:07] <Coren>	 Unless you expect that ops go crazy and take an ax to the actual gear.  :-)
[21:26:35] <valhallasw>	 Coren: and then some commercial operator takes over the infrastructure, and does not publish any dumps, and blocks any bot that tries to copy large-ish amounts of data
[21:26:59] <valhallasw>	 it's not very likely
[21:27:11] <Coren>	 valhallasw: Then you're simply back to the original solution; take that last dumps and build a new infrastructure to host it.
[21:27:30] <valhallasw>	 hoping that *someone* actually has a mirror of those dumps
[21:27:48] <valhallasw>	 but I think someone was doing that by now
[21:28:43] <Coren>	 valhallasw: I know there have been a couple sucessful dump restores on private servers in the past, to "test" the process.  They wouldn't have been able to cope with the wm traffic levels without a lot of $, but it's been done.
[21:29:08] <valhallasw>	 Coren: the point is not whether you can restore, but if there is a dump outside WMF property ;-)
[21:30:32] <Coren>	 valhallasw: Well, since anyone can take them without saying so, we just don't know.  But it's also not a worry; even if the WMF went boom, they wouldn't do so overnight and there'd be plenty of time to react.  Besides, do you sincerely think that the WMF people would make certain at least one good final dump of everything existed before they turned off the lights?
[21:31:10] <Coren>	 I know I sure as hell would make sure that my git clones of every bit of infrastructure config were up to date.
[21:31:33] <valhallasw>	 Coren: sure, a bankrupcy is forseeable
[21:33:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 546370.000000  
[21:34:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[21:35:58] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[21:37:13] <Coren>	 What else is there then?  Technical disaster?  Insured against, and while the current backup site is subpar it'd hold well enough until things got back on their feet.  Hostile maneuver?  Even if you could find *one* trustee willing to do something like this, you couldn't get a majority if your life depended on it.  And it'd need collaboration from most of the directors - which would 
[21:37:17] <Coren>	 also never happen.
[21:37:59] <Coren>	 And you're not even taking into account that the entire staff would raise a stink up at the first sign of something this gross.
[21:38:03] <valhallasw>	 Coren: government intervention
[21:38:16] <valhallasw>	 is the only thing big enough I can think of :-)
[21:38:47] <Susan>	 <Coren> No, it isn't, because it's neither rational nor well-founded; the efforts being deployed to provide a stable home for community maintained tools is large, and the result is strong.  You are rejecting it out of hand without a second glance because you dislike the WMF and not because of any characteristic of the Tool Labs /itself/
[21:38:53] <Coren>	 valhallasw: That's marginally more likely, but unlikely in the current political climate.  If I were the WMF, I'd make sure I had at least one datacenter in a different jurisdiction, though, just to make sure.
[21:39:03] <Susan>	 It's difficult to argue with you given that you're a contractor for Wikimedia.
[21:39:16] <Susan>	 It makes a dialogue difficult, I think.
[21:39:27] <Coren>	 Susan: Has it occured to you that I accepted the job because I *agreed* with what the job was?  :-)
[21:39:40] <Susan>	 Maybe.
[21:39:53] <Susan>	 I think you know that putting all of your eggs in one basket is bad practice.
[21:39:59] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13444.000000  
[21:40:00] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13444  
[21:40:02] <Susan>	 Security through segregation.
[21:40:08] <valhallasw>	 Coren: remember that because WMF is a US entity, the US will claim jurisdiction, irrespective of where the actual servers are.
[21:40:10] <Susan>	 Though any setup likely has issues.
[21:40:23] <Susan>	 Even a Toolserver still relies on the Wikimedia API, for example.
[21:40:28] <Susan>	 It's turtles all the way down.
[21:40:40] <Coren>	 Susan: Nope.  Four elephants at the bottom.  :-)
[21:40:58] <Susan>	 Labs makes me uncomfortable.
[21:41:09] <Coren>	 Wait, no, the elephants are on the turtles.  Nevermind.
[21:41:44] <Susan>	 I personally rely heavily on DB replication. But for others who just need a home for their small scripts, I think it would be easy enough to provide several homes.
[21:41:49] <Susan>	 And I think several homes are better than one sometimes.
[21:42:03] <Susan>	 You wouldn't put every person in the same house, for example.
[21:42:12] <Susan>	 Or even the same building or city. ;-)
[21:42:27] <Coren>	 Susan: Except that you multiply the resources (technical and human) needed; and you end up running out of what is, in the end, a finite resource.
[21:42:59] <Susan>	 As I said, most TS users use very few resources.
[21:43:12] <Susan>	 You could likely support 70% or more without much trouble.
[21:43:20] <Susan>	 It's people like me who cause the problems. ;-)
[21:44:44] <Coren>	 Susan: The enemy isn't consumed resources, but time.  Just keeping the infrastructure up to date is a serious endeavor.  Maintaining a VPS you are the only user to is simple enough; try it again after 5 years of users going in and out, hardware failiures, changes in venue.  :-)
[21:45:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[21:45:52] <Coren>	 Susan: The WMF /has/ a maintained datacenter, with good virtualization resources and a full time ops staff.  The overhead to had more compute nodes to provide the Labs is (comparatively) neglectable, and the infrastructure is already being maintained.
[21:46:09] <Coren>	 s/to had/to add/
[21:47:43] <Susan>	 Coren: Right, so I wouldn't change venue.
[21:47:45] <Coren>	 valhallasw: There is little to be done to cure the US from its imperialist tendency, I'm afraid.
[21:47:49] <Susan>	 And we'd obviously try to avoid hardware failure.
[21:47:57] <Susan>	 And every few years you'd need to update, of course.
[21:48:37] <Coren>	 Susan: I think you very much underestimate the amount of involved effort and resources.
[21:48:53] <Susan>	 Maybe.
[21:49:14] <Susan>	 I've run a host or two. I don't see many downsides against setting up a few more Toolserver hosts.
[21:49:20] <Susan>	 But it's just brainstorming for now.
[21:49:30] <Susan>	 If there are legitimate objections in the future, they can be addressed. :-)
[21:49:58] * Susan  --> dinner.

[21:50:12] <Coren>	 Good food acquisition.
[21:50:48] <Susan>	 Thank you. :-)
[21:50:53] <Susan>	 I'm meeting with Kirill and James and some others.
[21:51:10] <Coren>	 Ah.  I didn't even know you were in the UK
[21:51:18] <Susan>	 DC ;-)
[21:51:23] <Susan>	 Hare, not Forrester.
[21:51:26] <Coren>	 Ah!
[21:51:31] * Coren  waves.

[21:51:36] <Coren>	 Good meetup as well, then.
[22:01:29] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[22:03:58] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[22:06:09] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[22:07:19] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70476 MB (11% inode=99%):  
[22:07:28] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[22:08:38] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[22:08:39] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[22:08:39] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 23737 MB (2% inode=99%):  
[22:09:08] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[22:09:08] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 624763.000000  
[22:09:29] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340323 MB (6% inode=63%):  
[22:09:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[22:09:59] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 546038.000000  
[22:10:19] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 358732.000000  
[22:10:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[22:10:48] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[22:10:48] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[22:10:48] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[22:12:00] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[22:12:29] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[22:12:48] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[22:13:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[22:13:29] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[22:13:29] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[22:13:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:14:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:14:08] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:08] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:09] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:19] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[22:15:19] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:19] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:39] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[22:15:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[22:15:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[22:15:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:59] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:15:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:16:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:16:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[22:17:00] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[22:17:00] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[22:17:00] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[22:17:08] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:17:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:17:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:17:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:17:09] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:17:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:19:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[22:19:28] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:19:28] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:19:38] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:19:38] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[22:19:38] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[22:33:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 548568.000000  
[22:34:28] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[22:35:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[22:39:59] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14040.000000  
[22:40:59] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14003  
[22:45:39] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).  
[23:03:59] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:
[23:04:48] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2192.000000  
[23:04:48] <tsnag>	 s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2193.000000  
[23:06:08] <tsnag>	 SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[23:07:29] <tsnag>	 /mnt on thyme is UNKNOWN: NRPE: Unable to read output  
[23:08:19] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70277 MB (11% inode=99%):  
[23:09:28] <tsnag>	 /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340228 MB (6% inode=63%):  
[23:09:40] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146)  
[23:09:40] <tsnag>	 Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output  
[23:09:59] <tsnag>	 FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk  
[23:10:09] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 549549.000000  
[23:10:09] <tsnag>	 NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[23:10:09] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 627904.000000  
[23:10:49] <tsnag>	 aliasd on yarrow is CRITICAL: Connection refused  
[23:10:49] <tsnag>	 Load avg. on thyme is UNKNOWN: NRPE: Unable to read output  
[23:10:49] <tsnag>	 NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[23:11:18] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 361445.000000  
[23:11:18] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[23:12:00] <tsnag>	 / on thyme is UNKNOWN: NRPE: Unable to read output  
[23:12:29] <tsnag>	 /tmp on thyme is UNKNOWN: NRPE: Unable to read output  
[23:13:08] <tsnag>	 RAID on thyme is UNKNOWN: NRPE: Unable to read output  
[23:13:49] <tsnag>	 FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk  
[23:13:49] <tsnag>	 FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:14:08] <tsnag>	 FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:14:09] <tsnag>	 FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:14:28] <tsnag>	 / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[23:14:28] <tsnag>	 Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[23:15:08] <tsnag>	 FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:15:09] <tsnag>	 FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:15:29] <tsnag>	 FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:15:39] <tsnag>	 /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[23:15:49] <tsnag>	 FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:15:49] <tsnag>	 Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[23:15:49] <tsnag>	 FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:15:49] <tsnag>	 FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:15:59] <tsnag>	 Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[23:15:59] <tsnag>	 FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:15:59] <tsnag>	 FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:15:59] <tsnag>	 eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:16:09] <tsnag>	 FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:16:19] <tsnag>	 Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[23:16:19] <tsnag>	 FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:16:19] <tsnag>	 FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:16:38] <tsnag>	 aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[23:17:09] <tsnag>	 FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:17:09] <tsnag>	 FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:17:09] <tsnag>	 FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:17:09] <tsnag>	 FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:17:09] <tsnag>	 FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:17:38] <tsnag>	 FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:17:59] <tsnag>	 Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[23:17:59] <tsnag>	 SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[23:17:59] <tsnag>	 NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[23:19:29] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[23:19:29] <tsnag>	 FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:19:29] <tsnag>	 FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:19:39] <tsnag>	 FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:19:39] <tsnag>	 FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.  
[23:19:39] <tsnag>	 APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon.  
[23:28:48] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3633.000000  
[23:28:48] <tsnag>	 s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3633.000000  
[23:31:19] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[23:33:09] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 552168.000000  
[23:34:29] <tsnag>	 APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates).  
[23:36:59] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[23:39:59] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12309.000000  
[23:40:58] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12257  
[23:46:38] <tsnag>	 APT on sage is WARNING: APT WARNING: 3 packages available for upgrade (0 critical updates).