[00:05:51] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [00:06:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:08:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:09:51] CAM on hemlock is OK: OK - cam detected no new errors [00:13:06] Hello, I'm having some issues with CommonsHelper2. "No data received." Is there anything that can be done? [00:14:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 690384.000000 [00:19:11] Free Memory on damiana is WARNING: WARNING - 6.0% (502556 kB) free! [00:27:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (388488 kB) free! [00:34:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18489.000000 [00:35:50] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [00:39:51] CAM on hemlock is OK: OK - cam detected no new errors [00:40:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:42:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:43:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1192759.000000 [00:43:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32085 MB (5% inode=99%): [00:43:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 509856.000000 [00:43:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 812138.000000 [00:43:52] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:43:52] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:52] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:44:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:41] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:46:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:22] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:47:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:22] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:31] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:49:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:21] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:49:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:31] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:31] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:52] / on thyme is UNKNOWN: NRPE: Unable to read output [00:50:01] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:50:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:50:02] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:02] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 133 MB (0% inode=93%): [00:50:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:02] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:03] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:10] wikidata replag on daphne is CRITICAL: (Service Check Timed Out) [00:50:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:51:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:52:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:53:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 145 MB (0% inode=93%): [00:53:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:53:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:53:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:53:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:53:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:55:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 334463 MB (6% inode=61%): [00:55:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:04:52] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2172.000000 [01:05:52] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [01:06:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:09:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:09:52] CAM on hemlock is OK: OK - cam detected no new errors [01:11:31] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2201.000000 [01:15:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 693194.000000 [01:28:11] Free Memory on damiana is CRITICAL: CRITICAL - 1.6% (131600 kB) free! [01:28:51] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [01:34:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22090.000000 [01:35:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3641.000000 [01:35:51] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [01:39:51] CAM on hemlock is OK: OK - cam detected no new errors [01:40:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:42:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:43:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:43:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1196358.000000 [01:44:50] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32027 MB (5% inode=99%): [01:44:50] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 513516.000000 [01:44:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 815798.000000 [01:44:51] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:44:51] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:51] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:42] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:52] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:46:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:21] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:48:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:22] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:49:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:49:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:21] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:49:30] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:49:51] / on thyme is UNKNOWN: NRPE: Unable to read output [01:50:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:50:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:50:02] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:02] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 23 MB (0% inode=93%): [01:50:03] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:03] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:03] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:50:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:32] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1353380.000000 [01:51:32] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:52:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:52:42] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:53:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 31 MB (0% inode=93%): [01:53:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:53:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:53:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:53:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:53:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:54:31] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3556.000000 [01:54:51] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3257.000000 [01:55:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 334378 MB (6% inode=61%): [01:55:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:55:51] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [01:56:51] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1378.000000 [01:57:32] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1431.000000 [02:07:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:10:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:18:32] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 695124.000000 [02:19:19] Free Memory on damiana is OK: OK - 83.4% (6992092 kB) free. [02:35:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25748.000000 [02:40:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:42:35] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:00] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1196768.000000 [02:44:01] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:45:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 817340.000000 [02:45:14] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 515208.000000 [02:45:14] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32415 MB (5% inode=99%): [02:45:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:36] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:46:50] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:51] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:51] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:51] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:51] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:51] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:38] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:48:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:42] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:49:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:49:37] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:37] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:49:37] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:49:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:50:09] / on thyme is UNKNOWN: NRPE: Unable to read output [02:50:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:50:10] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:50:14] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:50:27] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [02:50:36] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:50:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:50:36] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:50:36] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:50:36] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:36] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:36] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:50:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:51:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:51:36] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:51:56] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2203.000000 [02:51:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1356867.000000 [02:52:11] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2212.000000 [02:52:12] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2212.000000 [02:52:12] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2212.000000 [02:52:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:53:11] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2266.000000 [02:53:11] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2288.000000 [02:53:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:53:29] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [02:53:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:53:35] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:53:35] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:53:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:55:08] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:55:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 334260 MB (6% inode=61%): [03:07:35] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:11:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:15:08] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [03:15:15] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [03:16:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [03:16:08] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 910.000000 [03:16:08] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3653.000000 [03:16:14] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3454.000000 [03:19:09] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 698826.000000 [03:21:14] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1691.000000 [03:21:29] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1819 [03:21:54] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2355 [03:21:55] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3010 [03:21:55] aliasd on yarrow is CRITICAL: Connection refused [03:22:09] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1600.000000 [03:22:09] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3003 [03:22:27] MySQL slave on z-dat-s5-b is OK: Uptime: 159614 Threads: 5 Questions: 289311469 Slow queries: 1708 Opens: 140408 Flush tables: 1 Open tables: 256 Queries per second avg: 1812.569 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1524 [03:23:55] MySQL slave on z-dat-s2-b is OK: Uptime: 246523 Threads: 15 Questions: 226386423 Slow queries: 3263 Opens: 2215202 Flush tables: 1 Open tables: 256 Queries per second avg: 918.317 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1416 [03:30:55] MySQL slave on z-dat-s1-b is OK: Uptime: 160823 Threads: 20 Questions: 133384163 Slow queries: 2190 Opens: 1062 Flush tables: 1 Open tables: 397 Queries per second avg: 829.384 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1798 [03:30:55] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1754.000000 [03:31:08] MySQL slave on rosemary is OK: Uptime: 25113 Threads: 11 Questions: 17295590 Slow queries: 2234 Opens: 1810 Flush tables: 1 Open tables: 524 Queries per second avg: 688.710 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1725 [03:36:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29377.000000 [03:40:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:42:35] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1200380.000000 [03:44:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:45:36] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:55] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:45:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 821012.000000 [03:46:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518881.000000 [03:46:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31865 MB (5% inode=99%): [03:46:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:47:37] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:37] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:37] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:37] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:44] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:44] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:44] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:49:35] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:50:08] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:50:27] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 708 MB (2% inode=93%): [03:50:27] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:50:36] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:50:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:50:36] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:37] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:37] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:37] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:50:37] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:50:38] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:50:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:50:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:51:08] / on thyme is UNKNOWN: NRPE: Unable to read output [03:51:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:51:09] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:51:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:51:35] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:51:45] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:51:45] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:52:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:52:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1360194.000000 [03:53:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:53:37] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:53:37] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:53:44] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:53:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:54:26] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 719 MB (2% inode=93%): [03:55:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:55:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 334084 MB (6% inode=61%): [04:12:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:16:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7245.000000 [04:16:08] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7253.000000 [04:20:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 702486.000000 [04:21:54] aliasd on yarrow is CRITICAL: Connection refused [04:36:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32978.000000 [04:37:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:41:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:42:36] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:44:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1203980.000000 [04:44:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:45:36] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:55] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:45:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 824672.000000 [04:47:08] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 522541.000000 [04:47:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31818 MB (5% inode=99%): [04:47:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:47:36] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:37] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:37] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:37] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:45] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:44] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:49:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:50:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:50:26] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 578 MB (1% inode=93%): [04:50:26] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:50:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:50:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:50:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:36] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:07] / on thyme is UNKNOWN: NRPE: Unable to read output [04:51:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:51:08] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:51:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:51:36] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:36] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:36] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:51:36] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:51:36] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:51:36] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:51:36] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:51:44] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:51:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:52:44] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:26] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:53:37] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:37] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:37] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:44] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:53:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1363543.000000 [04:55:07] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:55:26] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 573 MB (1% inode=93%): [04:56:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:56:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 334006 MB (6% inode=61%): [05:12:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:16:09] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10845.000000 [05:16:09] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10853.000000 [05:20:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 706087.000000 [05:21:55] aliasd on yarrow is CRITICAL: Connection refused [05:36:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36578.000000 [05:41:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:42:36] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:44:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1207580.000000 [05:44:10] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:45:36] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:55] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:45:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 828272.000000 [05:47:10] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 526141.000000 [05:47:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31713 MB (5% inode=99%): [05:47:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:47:36] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:36] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:36] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:37] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:45] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:45] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:49:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:50:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:50:29] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 440 MB (1% inode=93%): [05:50:29] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [05:50:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:50:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:50:36] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:50:36] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:50:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:10] / on thyme is UNKNOWN: NRPE: Unable to read output [05:51:10] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:51:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:51:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:51:36] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:36] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:37] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:51:37] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:51:37] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:51:37] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:51:37] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:51:44] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:51:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:52:45] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:53:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:37] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:37] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1366731.000000 [05:53:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:30] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (1 error): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [05:55:08] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:55:28] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 431 MB (1% inode=93%): [05:56:15] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:56:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 333809 MB (6% inode=61%): [05:59:28] CAM on hemlock is OK: OK - cam detected no new errors [06:07:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:13:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:16:09] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14445.000000 [06:17:09] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14513.000000 [06:20:08] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 709686.000000 [06:21:56] aliasd on yarrow is CRITICAL: Connection refused [06:25:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [06:29:29] CAM on hemlock is OK: OK - cam detected no new errors [06:36:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40178.000000 [06:41:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:42:37] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1211240.000000 [06:45:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:45:28] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [06:45:36] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:56] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:45:56] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:47:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 831872.000000 [06:47:09] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 529741.000000 [06:47:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31568 MB (5% inode=99%): [06:47:09] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:38] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:47:38] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:38] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:38] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:38] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:38] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:45] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:44] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:45] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:49:37] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:50:08] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:50:27] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 287 MB (0% inode=93%): [06:50:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:38] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:38] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:50:38] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:10] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:51:10] / on thyme is UNKNOWN: NRPE: Unable to read output [06:51:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:51:28] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:37] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:38] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:38] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:38] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:39] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:45] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:52:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:52:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:52:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:53:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:53:37] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:53:37] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:53:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:53:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:53:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1369965.000000 [06:53:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:55:27] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 272 MB (0% inode=93%): [06:56:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:56:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 333570 MB (6% inode=61%): [07:02:55] aliasd on yarrow is OK: TCP OK - 0.019 second response time on port 984 [500 Not found.] [07:03:07] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 106211 MB (10% inode=99%): [07:07:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:13:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:16:35] /tmp on wolfsbane is OK: DISK OK - free space: / 15830 MB (52% inode=93%): [07:16:35] / on wolfsbane is OK: DISK OK - free space: / 15828 MB (52% inode=93%): [07:17:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12051.000000 [07:17:12] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11595.000000 [07:21:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 711939.000000 [07:26:11] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3292.000000 [07:29:12] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1716.000000 [07:31:04] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3306.000000 [07:35:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1315.000000 [07:37:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43838.000000 [07:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:43:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:46:04] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:46:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1205364.000000 [07:46:04] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:46:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:12] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 530320.000000 [07:47:13] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31381 MB (5% inode=99%): [07:47:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:48:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:05] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:05] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:05] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:05] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:05] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 834212.000000 [07:49:04] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:49:05] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:49:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:50:12] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:51:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:05] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:51:05] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:51:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:51:22] / on thyme is UNKNOWN: NRPE: Unable to read output [07:51:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:51:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:52:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:04] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:04] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:52:04] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:52:04] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:52:05] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:52:05] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:52:06] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:52:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:53:04] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:53:04] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:53:34] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:54:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1373283.000000 [07:54:04] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:54:04] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:54:04] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:54:04] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:54:22] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:55:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:56:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:56:23] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 333323 MB (6% inode=61%): [08:05:35] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [08:07:32] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:09:33] CAM on hemlock is OK: OK - cam detected no new errors [08:13:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:22:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 710287.000000 [08:25:34] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [08:37:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47438.000000 [08:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:43:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:04] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:04] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:46:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1196495.000000 [08:46:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:23] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:11] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31194 MB (5% inode=99%): [08:47:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 528602.000000 [08:47:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:49:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:04] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:05] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:05] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:05] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:05] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:05] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:06] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 833402.000000 [08:49:52] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:50:12] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:06] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:06] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:06] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:51:22] / on thyme is UNKNOWN: NRPE: Unable to read output [08:51:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:51:51] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:52:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:53:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:53:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:05] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:05] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:53:05] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:53:05] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:53:05] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:53:06] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:53:06] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:53:07] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:54:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1376641.000000 [08:54:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:54:05] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:54:05] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:54:05] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:54:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:54:34] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:56:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:56:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:56:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332973 MB (6% inode=61%): [09:07:32] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:14:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:23:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 708999.000000 [09:38:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51098.000000 [09:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:44:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:04] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:46:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1188976.000000 [09:46:23] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:13] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 527013.000000 [09:47:13] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30957 MB (5% inode=99%): [09:47:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:48:32] Load avg. on ptolemy is WARNING: WARNING - load average: 15.29, 14.42, 10.61 [09:49:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:05] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:49:05] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:49:33] Load avg. on ptolemy is OK: OK - load average: 13.07, 14.01, 10.71 [09:50:02] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:02] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:02] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:03] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:03] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 834743.000000 [09:50:50] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:51:12] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:51:23] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:51:23] / on thyme is UNKNOWN: NRPE: Unable to read output [09:51:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:51:51] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:52:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:04] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:04] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:52:04] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:52:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:54:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:54:05] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1378298.000000 [09:54:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:06] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:06] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:06] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:07] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:07] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:08] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:08] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:09] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:09] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:22] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:55:33] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:56:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:56:24] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:56:24] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332649 MB (6% inode=61%): [10:07:32] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:14:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:23:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 708609.000000 [10:27:22] Free Memory on turnera is WARNING: WARNING - 5.1% (429732 kB) free! [10:29:22] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (423068 kB) free! [10:31:23] Free Memory on turnera is OK: OK - 9.6% (800972 kB) free. [10:38:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54698.000000 [10:42:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:44:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:04] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:04] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:46:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1187565.000000 [10:47:11] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 526980.000000 [10:47:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30764 MB (5% inode=99%): [10:47:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:49:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:04] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:49:04] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:03] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:03] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:04] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 836013.000000 [10:50:51] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:51:12] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:51:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:51:51] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:52:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:05] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:52:05] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:52:23] / on thyme is UNKNOWN: NRPE: Unable to read output [10:52:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:52:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:54:21] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:03] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:55:03] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1380946.000000 [10:55:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:03] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:04] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:04] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:05] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:05] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:07] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:33] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:56:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:56:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:56:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 332269 MB (6% inode=61%): [10:57:23] Free Memory on turnera is WARNING: WARNING - 6.9% (582312 kB) free! [10:59:24] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (402828 kB) free! [11:00:22] Free Memory on turnera is WARNING: WARNING - 6.5% (541632 kB) free! [11:07:32] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:14:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:21:22] Free Memory on turnera is CRITICAL: CRITICAL - 4.0% (337600 kB) free! [11:24:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 708413.000000 [11:32:23] Free Memory on turnera is WARNING: WARNING - 5.3% (441736 kB) free! [11:33:22] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (399644 kB) free! [11:35:21] Free Memory on turnera is WARNING: WARNING - 5.1% (430108 kB) free! [11:38:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58298.000000 [11:43:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:44:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:05] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:46:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1185089.000000 [11:47:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:13] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30542 MB (4% inode=99%): [11:47:13] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 526205.000000 [11:47:23] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:23] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:48:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:49:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:04] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:04] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:04] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:51:04] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:05] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:05] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:05] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:05] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 838692.000000 [11:51:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:51:53] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:52:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:05] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:52:05] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:52:12] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:53:22] / on thyme is UNKNOWN: NRPE: Unable to read output [11:53:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:53:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:54:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:55:02] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:55:03] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1384056.000000 [11:55:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:03] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:03] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:04] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:04] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:05] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:06] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:06] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:07] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:55:24] Free Memory on turnera is CRITICAL: CRITICAL - 3.4% (282560 kB) free! [11:56:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:56:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:56:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331537 MB (6% inode=61%): [11:56:32] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:10:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (1 error): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [12:14:43] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [12:14:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:19:43] CAM on hemlock is OK: OK - cam detected no new errors [12:25:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 707795.000000 [12:36:22] Free Memory on turnera is WARNING: WARNING - 5.1% (430548 kB) free! [12:37:23] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (384340 kB) free! [12:37:32] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:38:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61899.000000 [12:43:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:44:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:05] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:47:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1182622.000000 [12:47:22] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:22] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:48:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:48:12] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 526188.000000 [12:48:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30356 MB (4% inode=99%): [12:49:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:05] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:05] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:04] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:51:05] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:05] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:05] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:05] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:05] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 839022.000000 [12:51:24] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:52:12] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:52:52] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:53:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:05] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:05] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:53:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:53:22] / on thyme is UNKNOWN: NRPE: Unable to read output [12:53:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:53:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:54:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:54:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:55:03] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:55:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:04] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:04] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:04] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:04] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:04] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:05] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:05] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:06] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1387187.000000 [12:55:07] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:07] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:57:13] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:57:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:57:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 331223 MB (6% inode=61%): [12:57:32] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:00:45] Hello all [13:15:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:25:32] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 709256.000000 [13:37:32] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:38:21] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (325436 kB) free! [13:39:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65559.000000 [13:44:03] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2184.000000 [13:44:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:44:22] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2197.000000 [13:45:02] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:23] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:24] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:48:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:48:04] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:04] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:48:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1184828.000000 [13:49:11] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 529237.000000 [13:49:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30164 MB (4% inode=99%): [13:50:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:03] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:05] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:51:05] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:05] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:05] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:05] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:05] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:12] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 842411.000000 [13:52:23] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:52:52] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:53:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:54:03] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:04] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:54:04] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:54:23] / on thyme is UNKNOWN: NRPE: Unable to read output [13:54:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:54:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:55:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:56:03] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:56:03] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1390816.000000 [13:56:04] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:04] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:04] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:04] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:05] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:05] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:06] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:07] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:07] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:08] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:57:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:57:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:57:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330709 MB (6% inode=61%): [13:58:32] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:59:04] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:59:12] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:59:13] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:59:13] FC 0/23 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:59:13] FC 0/22 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:59:14] FC 0/20 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:59:14] FC 0/17 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:59:51] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:04] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:05] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:05] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:05] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:05] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:08:21] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [14:09:20] Free Memory on turnera is OK: OK - 91.4% (7661592 kB) free. [14:16:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:26:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 712917.000000 [14:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69181.000000 [14:44:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:46:00] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:50] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:48:00] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:48:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1188455.000000 [14:48:30] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:48:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:49:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 532838.000000 [14:50:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29972 MB (4% inode=99%): [14:50:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:40] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:30] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:51:43] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:50] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:00] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 846052.000000 [14:53:20] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:53:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:54:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:54:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:54:50] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:20] / on thyme is UNKNOWN: NRPE: Unable to read output [14:55:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:55:20] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:56:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:56:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:56:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:56:20] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:56:30] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:56:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1394445.000000 [14:56:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:56:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:56:30] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:56:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:56:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:57:00] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:30] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:57:30] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 330265 MB (6% inode=61%): [14:57:50] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:59:29] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:00:00] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:20] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:20] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:30] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:30] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:45] @replag [15:00:46] legoktm: s1-rr-a-wd: 1w 21h 25m 12s [-]; s1-user-c: 19h 34m 27s [-]; s1-user-wd: 1w 6d 18h 19m 55s [-]; s2-rr: 23s [-]; s2-user: 23s [-]; s2-user-c: error; s2-user-wd: 1w 2d 19h 8m 51s [-]; s3-user: 19s [-] [15:00:47] legoktm: s3-user-wd: 1w 1d 19h 49m 52s [-]; s4-user-wd: 2w 2d 3h 25m 8s [-]; s5-rr-a-wd: 6d 4h 12m 18s [-]; s5-user-c: 1h 53m 10s [-]; s6-user-wd: 1w 2d 55m 21s [-]; s7-user-wd: 1w 1d 6h 36m 24s [-] [15:00:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:27] legoktm@nightshade:~$ mysql -h sql-s5 wikidatawiki_p [15:01:32] mysql> SELECT UNIX_TIMESTAMP() - UNIX_TIMESTAMP(MAX(rc_timestamp)) FROM recentchanges; [15:01:36] | 0 | [15:07:27] dbreps@nightshade:~/src/database-reports/wikidatawiki$ mysql -h wikidatawiki-p.rrdb.toolserver.org wikidatawiki_p [15:07:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:07:34] mysql> SELECT UNIX_TIMESTAMP() - UNIX_TIMESTAMP(MAX(rc_timestamp)) FROM recentchanges; [15:07:35] | 762971 | [15:08:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7221.000000 [15:08:20] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7238.000000 [15:08:28] legoktm: rc_timestamp is not a unix-timestamp [15:09:09] i just copied that code from the tswiki [15:09:15] https://wiki.toolserver.org/view/Replag [15:10:56] ok [15:12:15] is the replag on the two hosts different? [15:12:20] or am i just doing it wrong? [15:12:55] mysql> select max(rc_timestamp) from recentchanges; [15:12:58] legoktm: every server has its own copy of wikidata [15:12:59] | 20130418191054 | [15:13:02] ok [15:13:08] it is normal that the replag differs [15:13:18] but by 9 days? [15:13:53] differedn hardware, differnd load, differend other clusters [15:14:25] ok [15:15:06] I never expected that wikidata would produce so many chances [15:15:19] you're not the only one :P [15:16:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:20:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3247.000000 [15:21:19] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3301.000000 [15:26:19] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1494.000000 [15:26:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715532.000000 [15:27:00] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1613.000000 [15:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72781.000000 [15:44:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:46:00] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:50] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:59] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1187473.000000 [15:48:30] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:50] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:49:00] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:49:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 534443.000000 [15:50:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29780 MB (4% inode=99%): [15:50:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:40] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:51:30] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:51:40] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:51:54] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:00] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:49] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 845676.000000 [15:53:20] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:53:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:54:10] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:54:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:54:50] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:51] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:55:20] / on thyme is UNKNOWN: NRPE: Unable to read output [15:55:20] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:56:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:56:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [15:56:21] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:56:29] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:56:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1397506.000000 [15:56:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:56:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:56:30] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:56:39] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:56:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:57:00] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:30] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:57:30] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 329769 MB (6% inode=61%): [15:57:50] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:59:29] DaBPunkt, I haven't received my daily cron emails from nightshade. Is something up with the mail daemon? [15:59:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:00:00] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:21] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:21] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:30] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:30] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:52] Dispenser: I will look [16:02:04] DaB. * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [16:03:46] Dispenser: the mail-log of nightshade shows that at least 2 mails have leave nightshade for your gmail-adress. [16:03:55] Should I check the mail-sending host? [16:04:21] I get emails, just not from my daily cron scripts [16:05:07] Last one was on the 23rd, but the job seems to be running since the logs are updating [16:06:03] Dispenser: the job finish from time to time? [16:07:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:08:55] It ran this morning 6:00 to 8:26 UTC. Hold on while I find me SSH key. [16:11:51] From cronie: two daily script and one run on Friday -- no emails from any of them [16:13:21] I do receive email via mutt [16:16:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:17:41] DaBPunkt: i'm getting also a lot of emails from hawthorn, some jobs are in qw since 22 yesterday's [16:18:10] Alch: Dispenser gets to FEW mails [16:18:35] i must be receiving his mails, cause i'm getting too mails XD [16:24:25] Alchimista: I freed some resources, lets look if you jobs are starting now [16:26:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 714678.000000 [16:26:34] DaBPunkt: thanks. i've notice that some jobs where with arch=lx, i've moved some to *, and must try to move the rest [16:37:56] DaBPunkt: nvm, the mail's in my spam filter ;-) [16:38:40] :-) [16:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76381.000000 [16:44:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:45:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:50] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:59] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1180985.000000 [16:48:30] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:50] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:49:00] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:49:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 534255.000000 [16:50:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29490 MB (4% inode=99%): [16:50:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:40] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:51:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:51:40] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:51:50] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:04] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 843915.000000 [16:53:20] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:53:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:54:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:10] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:50] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:55:21] / on thyme is UNKNOWN: NRPE: Unable to read output [16:55:21] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:56:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:56:20] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:30] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1400528.000000 [16:56:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:30] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:56:49] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:59] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:57:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 329384 MB (6% inode=61%): [16:57:50] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:59:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:00:00] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:00:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:00:20] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:00:21] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:00:29] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:00:30] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:00:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:07:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:16:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:26:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713535.000000 [17:39:29] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79982.000000 [17:44:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:46:00] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:50] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:48:00] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:48:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1173827.000000 [17:48:31] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:50] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:49:00] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:49:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 534960.000000 [17:49:59] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29206 MB (4% inode=99%): [17:50:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:40] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:51:30] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:51:40] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:51:49] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:04] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 842008.000000 [17:53:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:53:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:54:10] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:50] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:19] / on thyme is UNKNOWN: NRPE: Unable to read output [17:55:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:55:21] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:56:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:56:20] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:30] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1403754.000000 [17:56:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:30] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:56:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:57:00] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:01] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:30] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:57:31] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 326311 MB (6% inode=61%): [17:57:50] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:59:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:00:00] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:00:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:00:20] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:00:20] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:00:30] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:00:30] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:00:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:07:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:16:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:26:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 712828.000000 [18:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83581.000000 [18:44:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:45:59] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:00] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1168127.000000 [18:48:30] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:50] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:49:00] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:49:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 535618.000000 [18:50:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28948 MB (4% inode=99%): [18:50:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:40] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:51:40] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:50] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:03] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:49] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 842304.000000 [18:53:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:53:40] Free Memory on damiana is WARNING: WARNING - 6.0% (503904 kB) free! [18:53:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:54:10] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:20] / on thyme is UNKNOWN: NRPE: Unable to read output [18:55:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:55:21] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:56:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:56:20] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:29] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1407047.000000 [18:56:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:39] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:56:49] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:57:00] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:31] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:57:31] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 328271 MB (6% inode=61%): [18:57:40] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420604 kB) free! [18:57:50] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:58:40] Free Memory on damiana is WARNING: WARNING - 5.1% (431268 kB) free! [18:59:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:00:00] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:00:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:00:20] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:00:21] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:00:30] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:00:30] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:00:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:03:10] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 105495 MB (10% inode=99%): [19:09:40] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (340072 kB) free! [19:16:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:21:39] Free Memory on damiana is WARNING: WARNING - 7.0% (590676 kB) free! [19:26:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 712376.000000 [19:39:29] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87182.000000 [19:44:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:46:00] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:50] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:00] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:48:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1164135.000000 [19:48:30] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:50] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:49:00] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:49:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 535854.000000 [19:49:59] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28749 MB (4% inode=99%): [19:50:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:40] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:30] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:51:40] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:50] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:52:00] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:52:29] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:52:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 842352.000000 [19:53:20] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:53:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:54:10] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:50] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:20] / on thyme is UNKNOWN: NRPE: Unable to read output [19:55:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:55:20] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:56:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:56:30] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1410394.000000 [19:56:30] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:40] Free Memory on damiana is CRITICAL: CRITICAL - 2.6% (220816 kB) free! [19:56:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:56:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:57:00] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:20] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:57:31] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:57:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 327827 MB (6% inode=61%): [19:57:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:59:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:00:00] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:20] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:40] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:40] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:00:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:01:20] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:26:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 711563.000000 [20:37:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90782.000000 [20:44:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:44:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:46:00] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:50] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:00] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1161012.000000 [20:48:30] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:49:00] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:49:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 536119.000000 [20:50:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28454 MB (4% inode=99%): [20:50:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:39] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:30] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:51:43] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:50] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:52:00] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:52:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:52:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 841358.000000 [20:53:20] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:53:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:54:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:54:11] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:54:39] Free Memory on damiana is WARNING: WARNING - 5.6% (469624 kB) free! [20:54:50] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:20] / on thyme is UNKNOWN: NRPE: Unable to read output [20:56:09] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:56:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:56:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:56:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:56:21] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:56:30] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:56:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1413645.000000 [20:56:30] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:56:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:56:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:56:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:56:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:57:00] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:00] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:20] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:57:30] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:57:30] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 327251 MB (6% inode=61%): [20:57:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:59:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:00:00] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:20] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:00:40] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (411496 kB) free! [21:00:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:01:20] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:01:30] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:01:31] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:26:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 711213.000000 [21:37:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94383.000000 [21:44:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:44:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:46:00] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:50] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:00] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1160248.000000 [21:48:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:50] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:00] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:49:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 537359.000000 [21:49:59] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28180 MB (4% inode=99%): [21:50:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:40] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:39] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:51:40] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:54] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 842199.000000 [21:53:20] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:53:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:54:10] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:54:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:54:50] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:54:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:20] / on thyme is UNKNOWN: NRPE: Unable to read output [21:55:29] if one were to lose a TUSC password, would there be any way to retrieve such? [21:55:56] ... my friend... has lost her password [21:56:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:56:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:56:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:56:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:56:21] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:56:30] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:56:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1417040.000000 [21:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:56:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:56:40] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:56:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:56:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:57:00] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:01] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:57:30] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:57:30] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 326921 MB (6% inode=61%): [21:57:50] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:59:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:00:00] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:20] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:00:40] Free Memory on damiana is CRITICAL: CRITICAL - 1.6% (136136 kB) free! [22:00:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:21] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:40] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:40] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:01:54] i'm gonna assume no human beings say anything in here [22:02:00] * moogsi sidles up to tsnag [22:02:07] so, how YOU doin [22:16:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:26:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 711836.000000 [22:32:00] Sun Grid Engine execd on ortelius is CRITICAL: (Service Check Timed Out) [22:33:43] /mnt on thyme is CRITICAL: (Service Check Timed Out) [22:38:17] Free Memory on damiana is OK: OK - 59.4% (4977348 kB) free. [22:38:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:40:16] s4 replag on rosemary is CRITICAL: (Service Check Timed Out) [22:43:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:16] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 105261 MB (10% inode=99%): [22:44:46] / on thyme is UNKNOWN: NRPE: Unable to read output [22:44:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:45:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:45:26] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [22:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:46:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:46:25] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:47:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:16] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:16] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:26] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:36] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:36] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:36] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:36] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:36] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:37] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:37] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:39] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:39] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:40] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:48:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:46] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:49:16] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1160911.000000 [22:49:26] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:50:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 539193.000000 [22:50:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:06] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27914 MB (4% inode=99%): [22:52:16] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:53:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 843144.000000 [22:55:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:56:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:57:16] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:57:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1420137.000000 [22:57:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:56] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 326705 MB (6% inode=61%): [22:58:06] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:13:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:27:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713120.000000 [23:37:40] nacht ts [23:39:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:40:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101638.000000 [23:43:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:44:46] / on thyme is UNKNOWN: NRPE: Unable to read output [23:44:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:45:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:45:26] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [23:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:46:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:46:26] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:16] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:16] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:26] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:36] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:36] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:36] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:36] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:36] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:36] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:37] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:37] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:38] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:39] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:39] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:40] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:48:35] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:36] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:45] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:49:16] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1162116.000000 [23:49:26] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:50:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 541537.000000 [23:50:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:06] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27664 MB (4% inode=99%): [23:52:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:53:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 845041.000000 [23:55:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:56:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:57:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:57:16] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:57:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:57:27] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:57:35] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1423319.000000 [23:57:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325251 MB (6% inode=61%): [23:58:06] /tmp on thyme is UNKNOWN: NRPE: Unable to read output