[00:00:50] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 318 MB (1% inode=92%): [00:03:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:05:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:06:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:10:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317370 MB (5% inode=62%): [00:13:48] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:13:49] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:14:27] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:14:29] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:15:20] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:16:51] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:16:53] SSH on mayapple is CRITICAL: Server answer: [00:16:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:17:59] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [00:18:00] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [00:21:56] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:29] Danny_B: Do you have some dumps copying at the moment? [00:26:18] let me see [00:26:45] no script running, why? [00:28:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321463 MB (6% inode=62%): [00:29:02] scfc_de: ^^ [00:29:27] NFS seems to be rather slow, and dumps is the only thing that came to my mind :-). [00:30:06] Never mind. [00:33:39] scfc_de: dumps now run via /tmp [00:34:03] so the only touching of nfs is when copying from tmp to userstore [00:36:33] otoh i'll have to run unpack of enwiki which can't be done on tmp because of low space there [00:39:44] Danny_B: Yeah, I remember some discussion about that. Can I see somewhere if the scripts are running? [00:40:01] which scripts? [00:41:07] btw nfs is very unstable recently [00:41:43] every day i become disconnected several times [00:44:12] Oh, you unpack manually? I thought where was a bot or something. [00:44:19] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:44:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:44:57] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:57] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:57] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:44:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:57] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:58] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:45:18] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:18] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:48] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:49] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:49] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:49] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:45:57] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:57] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:57] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:45:57] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:46:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 677434.000000 [00:46:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 894672.000000 [00:46:27] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:27] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:46:48] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:48] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:58] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:47:13] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:47:32] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:47:38] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1285356.000000 [00:47:55] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:48:02] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:48:02] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:03] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:03] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:49] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 449645.000000 [00:48:50] / on thyme is UNKNOWN: NRPE: Unable to read output [00:48:50] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57226 MB (9% inode=99%): [00:49:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1191109.000000 [00:49:12] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:13] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:49:13] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:49:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:13] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:39] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:49:39] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:49:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:39] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:39] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:55] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:49:55] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:49:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:50:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:50:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:50:20] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:50:20] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:38] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 710655.000000 [00:50:39] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:50:39] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:50:48] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:50:48] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:48] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:50:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:51:21] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:51:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1990277.000000 [00:51:38] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:38] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:52] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:52] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:52:16] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:52:16] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:52:16] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:52:49] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:55:08] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [00:58:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:01:16] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 267 MB (0% inode=92%): [01:04:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:05:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:06:42] scfc_de: only enwiki. others are done automatically [01:07:17] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:10:06] Danny_B: And on which server? As user dumps? [01:10:21] willow [01:11:56] Ah, okay, so there is a process running now. [01:12:33] yes as i said before - unpacking the enwiki [01:12:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318039 MB (5% inode=62%): [01:12:48] but i started after we started to talk [01:13:22] Okay. [01:13:49] and it won't run long [01:14:07] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:14:17] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:14:47] No problem, NFS was far from unusable before. What's gcp? [01:15:16] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:15:17] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:15:36] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:06] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:18] SSH on mayapple is CRITICAL: Server answer: [01:18:06] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [01:18:06] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [01:22:21] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:47] / on willow is WARNING: DISK WARNING - free space: / 16877 MB (16% inode=98%): [01:37:47] /tmp on willow is WARNING: DISK WARNING - free space: / 16877 MB (16% inode=98%): [01:44:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:44:47] / on willow is CRITICAL: DISK CRITICAL - free space: / 10991 MB (10% inode=98%): [01:44:47] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 10991 MB (10% inode=98%): [01:45:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:45:17] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:17] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:18] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:18] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:18] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:18] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:45:47] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:47] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:45:56] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:16] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:46:16] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:16] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:17] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:46:17] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:46:17] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:46:17] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:46:18] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:46:18] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 892645.000000 [01:46:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 674597.000000 [01:46:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:46:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:48] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:07] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:17] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:17] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:47:37] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1288724.000000 [01:47:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:16] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:48:16] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:48:16] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:16] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:17] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:26] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:49:17] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:17] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:49:17] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:49:17] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:49:17] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:49:18] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:49:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:49:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:49:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:36] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:36] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:47] / on thyme is UNKNOWN: NRPE: Unable to read output [01:49:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 449248.000000 [01:49:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56197 MB (9% inode=99%): [01:50:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1191665.000000 [01:50:47] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:50:47] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:50:47] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:50:48] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:50:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:50:48] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:50:48] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:50:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:50:50] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:51:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:51:17] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:51:17] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:51:17] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:51:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1993649.000000 [01:51:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 714315.000000 [01:52:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:52:18] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:52:18] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:52:18] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:52:36] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:36] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:47] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:48] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:53:47] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:56:07] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [01:58:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:01:17] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 183 MB (0% inode=92%): [02:04:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:05:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:06:48] / on willow is OK: DISK OK - free space: / 26371 MB (25% inode=99%): [02:06:48] /tmp on willow is OK: DISK OK - free space: / 26371 MB (25% inode=99%): [02:07:17] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:10:47] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324327 MB (6% inode=62%): [02:14:07] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:14:17] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:17] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:17] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:37] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:17:07] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:18:06] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [02:18:06] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [02:18:16] SSH on mayapple is CRITICAL: Server answer: [02:22:17] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:42:17] Load avg. on nightshade is WARNING: WARNING - load average: 18.93, 18.02, 10.61 [02:43:06] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2163 [02:43:07] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2163.000000 [02:44:17] Load avg. on nightshade is OK: OK - load average: 4.71, 12.90, 9.63 [02:44:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:45:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:45:17] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:17] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:17] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:45:47] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:47] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:45:56] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:16] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:16] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:16] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:16] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:17] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:17] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:17] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:18] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:18] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:46:19] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:19] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:20] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 889727.000000 [02:46:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:46:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:48] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:07] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:17] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:17] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:47:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 671860.000000 [02:47:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1291657.000000 [02:47:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:17] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:48:17] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:48:17] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:17] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:18] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:26] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:49:17] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:17] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:49:17] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:49:17] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:49:17] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:49:18] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:49:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:49:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:49:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:36] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:36] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:47] / on thyme is UNKNOWN: NRPE: Unable to read output [02:49:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 447222.000000 [02:49:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57036 MB (9% inode=99%): [02:50:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1192718.000000 [02:50:47] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:51:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:51:16] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:51:17] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:51:17] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:51:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1997087.000000 [02:51:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 717915.000000 [02:51:48] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:51:48] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:51:48] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:51:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:51:48] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:51:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:51:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:51:50] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:51:50] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:52:16] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:52:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:52:47] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:47] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:53:16] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:53:16] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:53:36] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:36] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:48] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:56:07] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [02:58:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:01:17] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 122 MB (0% inode=92%): [03:04:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:05:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:07:16] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:09:07] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3636 [03:09:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [03:10:47] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 365700 MB (6% inode=62%): [03:14:06] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:14:16] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:15:18] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:15:18] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:15:37] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:16:48] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2127.000000 [03:17:08] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:18:09] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 908 MB (3% inode=92%): [03:18:09] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 908 MB (3% inode=92%): [03:18:17] SSH on mayapple is CRITICAL: Server answer: [03:22:18] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:42:36] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 90602 MB (9% inode=99%): [03:43:49] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1644.000000 [03:44:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:45:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:45:18] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:18] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:18] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:45:49] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:49] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:46:17] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:17] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:17] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:17] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:17] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:18] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:18] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:19] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:19] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:20] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:20] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:21] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 889194.000000 [03:46:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:46:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:57] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:08] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:18] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:18] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:47:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 669120.000000 [03:47:36] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:47:50] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1294711.000000 [03:48:27] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:49:17] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:17] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:49:18] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:49:18] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:49:18] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:49:18] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:49:18] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:49:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:49:19] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:49:20] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:49:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:49:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:49:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:48] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:49] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:49] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:49] / on thyme is UNKNOWN: NRPE: Unable to read output [03:49:49] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 444729.000000 [03:49:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56953 MB (9% inode=99%): [03:50:08] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1195806.000000 [03:50:49] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:51:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:51:18] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:51:18] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:51:18] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:51:27] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2000597.000000 [03:51:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 721515.000000 [03:51:48] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:51:48] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:51:48] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:51:48] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:51:48] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:51:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:51:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:51:50] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:51:50] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:52:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:52:18] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:52:49] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:53:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:53:17] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:53:38] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:38] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:50] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:56:08] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [03:57:08] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3343 [03:57:08] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3344.000000 [03:58:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:01:27] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 2337 MB (7% inode=92%): [04:04:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:05:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:07:17] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:09:07] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2064 [04:10:49] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 365576 MB (6% inode=62%): [04:12:08] MySQL slave on z-dat-s5-b is OK: Uptime: 285497 Threads: 5 Questions: 500286429 Slow queries: 3242 Opens: 166847 Flush tables: 1 Open tables: 256 Queries per second avg: 1752.335 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1657 [04:12:08] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1657.000000 [04:14:08] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:14:17] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:50] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:16:17] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:16:17] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:08] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:18:09] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 623 MB (2% inode=92%): [04:18:09] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 623 MB (2% inode=92%): [04:18:27] SSH on mayapple is CRITICAL: Server answer: [04:23:17] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:45:27] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:46:16] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:16] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:26] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:26] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:26] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:46:47] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:47] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:55] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:17] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:17] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:17] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:17] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:18] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:18] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:18] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:18] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:47:18] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 888435.000000 [04:47:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:47:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:48] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:06] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:48:16] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:16] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:48:27] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 666896.000000 [04:48:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:48] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1298210.000000 [04:49:26] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:49:26] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:49:27] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:49:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:49:37] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:49:37] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:49:37] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:16] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:16] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:50:16] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:50:16] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:50:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:50:17] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:50:17] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:50:49] / on thyme is UNKNOWN: NRPE: Unable to read output [04:50:49] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 439826.000000 [04:50:49] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:49] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:49] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:49] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:50:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56897 MB (9% inode=99%): [04:51:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1198955.000000 [04:51:26] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:51:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 725116.000000 [04:52:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:52:16] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:52:16] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:52:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2004146.000000 [04:52:48] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:52:48] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:52:48] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:52:48] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:52:48] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:52:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:52:48] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:52:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:52:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:53:16] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:16] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:16] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:16] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:53:47] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:47] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:54:36] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:54:36] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:47] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:57:06] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [04:59:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:01:27] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 2235 MB (7% inode=92%): [05:05:07] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3602 [05:05:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:06:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:08:16] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:11:47] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 365362 MB (6% inode=62%): [05:14:16] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:15:16] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:15:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:16:47] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:06] aliasd on yarrow is CRITICAL: Connection refused [05:17:17] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:17] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:17] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:19:07] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 391 MB (1% inode=92%): [05:19:07] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 391 MB (1% inode=92%): [05:19:31] SSH on mayapple is CRITICAL: Server answer: [05:23:17] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:06] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3532 [05:34:07] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3603 [05:36:06] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3594 [05:45:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:45:27] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:46:16] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:27] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:27] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:27] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:27] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:47] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:47] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:47:07] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:27] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:47:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:27] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:47:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 887889.000000 [05:48:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:07] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:16] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:16] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:16] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:16] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:16] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:48:27] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:48:27] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:48:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 664400.000000 [05:48:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:48:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:49:16] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:49:16] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:49:27] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:49:27] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:49:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:49:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:49:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:49:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1301437.000000 [05:50:16] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:50:16] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:50:16] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:50:16] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:50:17] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:50:17] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:50:17] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:50:35] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:50:36] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:50:36] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1201412.000000 [05:51:28] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:51:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 728716.000000 [05:51:47] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:51:47] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:51:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:51:47] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:51:47] / on thyme is UNKNOWN: NRPE: Unable to read output [05:51:48] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 431625.000000 [05:51:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56775 MB (9% inode=99%): [05:52:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:52:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2007618.000000 [05:52:56] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:53:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:16] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:53:16] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:53:31] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:47] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:53:47] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:53:47] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:53:47] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:47] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:53:48] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:53:48] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:54:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:54:15] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:54:16] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:54:27] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:54:36] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:54:37] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:55] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:56:07] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3779 [05:57:07] APT on yucca is WARNING: APT WARNING: 5 packages available for upgrade (0 critical updates). [05:59:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:01:26] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 2122 MB (7% inode=92%): [06:05:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:07:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:08:26] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:12:47] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 365101 MB (6% inode=62%): [06:13:07] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3512 [06:14:27] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:15:27] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:15:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:16:07] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3617 [06:16:47] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:17:07] aliasd on yarrow is CRITICAL: Connection refused [06:17:26] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:17:27] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:17:28] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:19:06] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 97 MB (0% inode=92%): [06:19:07] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 97 MB (0% inode=92%): [06:19:28] SSH on mayapple is CRITICAL: Server answer: [06:21:06] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:24:16] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:27] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7988&oldid=7986&rcid=21870 * Hazard-SJ * (+535) (/* …move somewhen this year. */ +) [06:31:07] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [06:31:07] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:31:16] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:31:26] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:32:22] SSH on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:11] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3594 [06:37:31] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:37:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:37:32] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:37:32] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:37:32] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:32] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:33] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:33] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:34] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:34] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:35] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:35] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:36] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:36] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:37:37] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:37:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:38:37] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3166 [06:45:32] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:45:34] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:46:34] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:34] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:34] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:34] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:34] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:53] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:53] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:47:35] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:35] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:54] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:54] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:54] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:39] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:48:39] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 885653.000000 [06:48:39] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:48:45] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:48:56] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:04] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:49:50] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 663328.000000 [06:49:50] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:49:51] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1304696.000000 [06:49:51] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:49:52] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:49:53] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:50:56] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:50:57] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:50:57] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:57] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:51:45] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:51:48] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 732327.000000 [06:51:48] / on thyme is UNKNOWN: NRPE: Unable to read output [06:51:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1204383.000000 [06:52:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:52:08] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:52:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 427930.000000 [06:52:46] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:52:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56661 MB (9% inode=99%): [06:52:46] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:52:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:53:04] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2011110.000000 [06:53:12] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:53:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:53:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:53:51] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:53:51] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:53:55] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:53:55] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:54:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:54:11] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:54:23] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:54:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:54:46] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:45] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:55:46] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:55:46] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:01] aliasd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:57:04] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:58:08] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:59:09] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [06:59:57] aliasd on nightshade is OK: TCP OK - 0.005 second response time on port 984 [500 Not found.] [06:59:57] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [06:59:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:02:00] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 2019 MB (6% inode=92%): [07:04:44] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3635 [07:05:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:08:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:09:44] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:14:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 364902 MB (6% inode=62%): [07:18:15] aliasd on yarrow is CRITICAL: Connection refused [07:18:15] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.771 second response time [07:18:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:18:18] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:26] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:32] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:32] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:34] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:19:52] SSH on mayapple is CRITICAL: Server answer: [07:19:52] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [07:19:53] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [07:19:53] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.518 second response time [07:19:53] Load avg. on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:09] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:21:09] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.153 second response time [07:23:52] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7989&oldid=7988&rcid=21871 * Nemobis * (+348) (/* …not move at all. */ +1) [07:25:57] SSH on z-dat-s5-b is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [07:25:59] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3507 [07:25:59] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:04] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7990&oldid=7989&rcid=21872 * Nemobis * (+104) (copy votes from list) [07:31:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [07:31:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:32:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:33:30] Load avg. on nightshade is WARNING: WARNING - load average: 6.38, 12.88, 19.29 [07:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:37:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:37:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:37:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:37:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:37:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:37:41] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:37:41] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:37:42] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:37:42] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:37:43] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:38:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:38:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:38:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:39:29] Load avg. on nightshade is OK: OK - load average: 4.66, 7.07, 14.53 [07:44:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:41] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:00] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:47:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:19] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:19] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:48:39] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 884216.000000 [07:48:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:48:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:49:40] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1307946.000000 [07:50:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 661659.000000 [07:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:50:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:50:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:50:40] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:51:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 735941.000000 [07:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 426523.000000 [07:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:52:40] / on thyme is UNKNOWN: NRPE: Unable to read output [07:52:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1207243.000000 [07:52:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:53:10] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2014563.000000 [07:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:53:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56426 MB (9% inode=99%): [07:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:54:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:54:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:54:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:54:40] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:54:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:54:40] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:54:41] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:54:41] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:55:39] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [08:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:02:10] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 1864 MB (6% inode=92%): [08:06:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:08:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:14:39] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 364648 MB (6% inode=62%): [08:18:41] aliasd on yarrow is CRITICAL: Connection refused [08:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:11] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:59] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [08:20:40] SSH on mayapple is CRITICAL: Server answer: [08:20:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [08:21:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:22:59] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:24:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:26:00] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:26:40] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2667 [08:26:40] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:28:40] Load avg. on nightshade is WARNING: WARNING - load average: 28.88, 22.05, 14.54 [08:28:49] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [08:29:30] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [08:31:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [08:31:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:32:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:32:29] Load avg. on nightshade is OK: OK - load average: 5.75, 14.03, 13.09 [08:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:37:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:37:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:37:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:37:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:37:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:37:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:37:41] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:37:42] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:37:42] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:37:49] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:38:19] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:38:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:38:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:38:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:46:20] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:46:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:47:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:47:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:47:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:19] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:39] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 883388.000000 [08:48:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:49:40] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:50:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1311368.000000 [08:50:39] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 660684.000000 [08:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:50:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:50:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739541.000000 [08:52:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:52:19] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 425227.000000 [08:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:52:40] / on thyme is UNKNOWN: NRPE: Unable to read output [08:52:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1210246.000000 [08:52:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:41] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:53:10] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2018025.000000 [08:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:53:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56205 MB (9% inode=99%): [08:54:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:30] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:54:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [08:54:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:54:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:54:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:54:41] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:54:41] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:54:50] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [09:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:02:10] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 1784 MB (5% inode=92%): [09:06:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:08:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:10:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:14:40] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 362682 MB (6% inode=62%): [09:18:40] aliasd on yarrow is CRITICAL: Connection refused [09:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:19:59] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [09:20:40] SSH on mayapple is CRITICAL: Server answer: [09:20:41] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [09:21:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:09] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.536 second response time [09:30:10] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.452 second response time [09:31:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [09:31:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:32:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:37:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:37:49] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:37:49] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:38:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:38:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:38:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:38:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:38:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:38:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:38:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:38:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:38:41] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:39:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:40:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [09:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:46:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:46:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:46:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:47:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:47:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:47:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:50] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 881905.000000 [09:48:41] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:49:46] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:49:46] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1314426.000000 [09:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 659161.000000 [09:50:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:50:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743141.000000 [09:52:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:52:19] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 424075.000000 [09:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:52:40] / on thyme is UNKNOWN: NRPE: Unable to read output [09:52:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1213455.000000 [09:52:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:53:10] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2021446.000000 [09:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55952 MB (9% inode=99%): [09:53:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:54:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5396 [09:54:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:54:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:54:41] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:54:50] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:55:39] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:10] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.428 second response time [09:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [10:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:02:10] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 1695 MB (5% inode=92%): [10:02:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:06:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:08:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:15:40] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 355342 MB (6% inode=62%): [10:18:40] aliasd on yarrow is CRITICAL: Connection refused [10:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:19:11] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:19:59] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [10:20:40] SSH on mayapple is CRITICAL: Server answer: [10:20:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [10:21:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [10:31:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [10:36:20] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:36:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:37:09] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:37:50] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:37:50] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:38:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:38:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:38:21] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:38:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:38:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:38:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:38:49] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:39:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:45:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [10:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:46:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:46:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:46:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:50] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:00] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:47:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:47:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:47:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:47:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 879313.000000 [10:48:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:49:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:49:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:49:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:49:40] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1317842.000000 [10:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 657411.000000 [10:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:50:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:39] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:51:50] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:52:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 746741.000000 [10:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 421023.000000 [10:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:52:40] / on thyme is UNKNOWN: NRPE: Unable to read output [10:52:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:49] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1216545.000000 [10:52:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:53:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2024866.000000 [10:53:09] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:53:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55759 MB (9% inode=99%): [10:53:41] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:54:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:54:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6514 [10:54:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:54:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:54:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:54:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [10:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [11:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:02:10] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 898 MB (2% inode=92%): [11:06:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:08:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:40] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 334493 MB (6% inode=62%): [11:18:40] aliasd on yarrow is CRITICAL: Connection refused [11:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:19:11] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:20:00] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [11:20:40] SSH on mayapple is CRITICAL: Server answer: [11:20:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [11:21:41] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:26:50] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [11:31:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [11:36:20] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:37:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:37:50] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:37:50] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:38:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:38:19] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:38:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:38:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:38:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:38:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:38:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:39:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:39:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:39:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:45:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [11:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:46:41] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:46:41] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:46:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:50] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:47:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:47:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 877978.000000 [11:48:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:49:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:49:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:49:40] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:50:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1321221.000000 [11:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 655640.000000 [11:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:50:49] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:50:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:51:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:51:50] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:52:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 750341.000000 [11:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 417367.000000 [11:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:52:40] / on thyme is UNKNOWN: NRPE: Unable to read output [11:52:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:53:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2028232.000000 [11:53:09] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:53:09] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:53:09] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:53:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1219530.000000 [11:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:45] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:54:45] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6947 [11:54:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55530 MB (9% inode=99%): [11:54:45] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:54:45] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:54:45] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:54:45] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:54:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:54:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [11:58:39] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319443 MB (5% inode=62%): [11:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [12:00:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:02:09] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [12:06:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:09:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:18:40] aliasd on yarrow is CRITICAL: Connection refused [12:19:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:19:09] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:19:09] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:19:09] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:19:09] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:20:00] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [12:20:40] SSH on mayapple is CRITICAL: Server answer: [12:20:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [12:21:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:26:50] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:40] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3505 [12:31:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [12:31:41] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [12:35:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:36:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:36:20] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:37:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:37:49] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:37:49] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:38:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:38:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:38:21] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:38:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:38:41] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:39:40] MySQL slave on z-dat-s2-b is OK: Uptime: 316010 Threads: 14 Questions: 327968448 Slow queries: 7762 Opens: 2659958 Flush tables: 1 Open tables: 256 Queries per second avg: 1037.841 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1791 [12:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:39:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:39:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:39:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:45:39] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:46:20] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:47:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:47:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:50] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 877816.000000 [12:48:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:49:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1324197.000000 [12:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 653947.000000 [12:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:50:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:50:40] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:50:49] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:20] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:51:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 753940.000000 [12:52:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:52:19] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 415187.000000 [12:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:52:41] / on thyme is UNKNOWN: NRPE: Unable to read output [12:52:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:53:10] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2031694.000000 [12:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:53:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1221756.000000 [12:54:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:54:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:54:41] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55332 MB (9% inode=99%): [12:54:41] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:54:41] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:54:41] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:54:41] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:54:50] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:54:50] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:57:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [12:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [12:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 298417 MB (5% inode=62%): [12:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [13:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:02:10] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [13:06:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:08:40] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2065.000000 [13:08:41] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2066 [13:09:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:10:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:18:39] aliasd on yarrow is CRITICAL: Connection refused [13:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:19:59] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [13:20:41] SSH on mayapple is CRITICAL: Server answer: [13:20:41] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [13:21:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [13:31:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [13:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:37:49] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:37:49] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:39:19] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:39:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:39:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:39:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:39:41] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:39:41] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:39:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:39:41] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:00] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:47:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:47:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:19] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:19] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 876984.000000 [13:48:43] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1326888.000000 [13:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 650841.000000 [13:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:51:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:51:40] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:51:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:52:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 757541.000000 [13:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 413807.000000 [13:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:52:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:53:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2034528.000000 [13:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [13:53:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1223044.000000 [13:53:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55113 MB (9% inode=99%): [13:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:54:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:54:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:41] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:55:41] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:55:41] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:55:41] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:57:09] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [13:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [13:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 277858 MB (5% inode=62%): [13:59:42] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3639.000000 [13:59:42] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3640 [13:59:42] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [14:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:02:09] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [14:06:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:10:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:16:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:18:40] aliasd on yarrow is CRITICAL: Connection refused [14:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:19:11] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:19:59] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [14:20:40] SSH on mayapple is CRITICAL: Server answer: [14:21:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [14:21:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [14:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:40] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3528.000000 [14:31:40] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3525 [14:31:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [14:31:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [14:36:20] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:37:50] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:39:19] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:39:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:39:19] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:39:39] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:39:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:39:41] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:40:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:40:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [14:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:50] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:47:00] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:47:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:47:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:47:41] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:47:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 875600.000000 [14:48:42] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:49:40] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000 [14:49:40] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628 [14:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1328619.000000 [14:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 646956.000000 [14:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:51:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:51:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 761140.000000 [14:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 412735.000000 [14:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:52:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:53:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2034186.000000 [14:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:53:39] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3514.000000 [14:53:40] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3511 [14:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [14:53:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1222924.000000 [14:53:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54836 MB (8% inode=99%): [14:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:54:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:54:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:41] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:55:41] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:55:41] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:55:41] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:56:11] @replag [14:56:11] FunPika: s1-rr-a-wd: 1w 4d 1m 59s [-]; s1-user-c: 1w 1d 19h 29m 53s [-]; s1-user-wd: 2w 1d 9h 6m 31s [-]; s2-user-c: error; s2-user-wd: 2w 3h 42m 20s [-]; s3-user-wd: 1w 3d 6h 5m 6s [-]; s4-user-wd: 3w 2d 13h 27s [-]; s5-rr-a-wd: 4d 18h 36m 50s [-] [14:56:12] FunPika: s5-user: 52m 1s [-]; s6-user-wd: 1w 3d 3h 11m 20s [-]; s7-user-wd: 1w 11h 38m 36s [-] [14:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:57:09] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [14:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [14:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 271496 MB (5% inode=62%): [14:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [15:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:02:10] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [15:06:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:09:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:15:40] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3631.000000 [15:15:40] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3632 [15:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:18:39] aliasd on yarrow is CRITICAL: Connection refused [15:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:19:59] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [15:20:40] SSH on mayapple is CRITICAL: Server answer: [15:21:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [15:21:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [15:32:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [15:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:37:49] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:37:49] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:38:19] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:39:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:39:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:39:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:39:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:39:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:39:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:40:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:40:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:42:40] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 91623 MB (9% inode=99%): [15:42:40] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3413.000000 [15:42:40] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3408 [15:45:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:46:08] I get bad gateway when connecting to https://toolserver.org/~acc/acc.php [15:46:20] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:47:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:47:41] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:47:41] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:47:41] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:47:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:47:50] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:00] Same here. [15:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 874275.000000 [15:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1329338.000000 [15:50:39] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 642345.000000 [15:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:51:20] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:51:40] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1581.000000 [15:51:40] MySQL slave on z-dat-s5-b is OK: Uptime: 327471 Threads: 3 Questions: 602175833 Slow queries: 3579 Opens: 206354 Flush tables: 1 Open tables: 256 Queries per second avg: 1838.867 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1576 [15:51:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:51:40] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:51:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 764741.000000 [15:52:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:52:19] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 411690.000000 [15:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:53:20] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2029435.000000 [15:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [15:53:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1223379.000000 [15:53:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:54:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54648 MB (8% inode=99%): [15:54:50] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:54:50] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:55:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:56:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:56:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:56:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:57:09] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [15:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [15:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 271330 MB (5% inode=62%): [15:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [16:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:02:10] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [16:06:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:09:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:10:09] 502 bad gateway on web [16:10:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:18:40] aliasd on yarrow is CRITICAL: Connection refused [16:19:09] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:19:59] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [16:20:40] SSH on mayapple is CRITICAL: Server answer: [16:21:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [16:22:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:26:50] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [16:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [16:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:37:49] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:37:49] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:38:19] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:38:19] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:38:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:39:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:39:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:39:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:39:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:39:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:39:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:39:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:40:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:40:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [16:45:39] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:47:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:47:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:47:50] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:48:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:49] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:49:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 873489.000000 [16:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:49:50] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1331444.000000 [16:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 638034.000000 [16:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:50:49] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:51:20] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 768341.000000 [16:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:52:19] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 410251.000000 [16:52:39] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:52:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:52:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:52:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:53:20] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2025980.000000 [16:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [16:53:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1224099.000000 [16:53:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:54:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:54:41] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54348 MB (8% inode=99%): [16:54:50] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:54:50] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:56:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:56:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:56:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:56:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [16:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 271208 MB (5% inode=62%): [16:59:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [17:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:02:10] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [17:06:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:07:10] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.969 second response time [17:08:09] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.693 second response time [17:09:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:10:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:11:10] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.435 second response time [17:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:18:40] aliasd on yarrow is CRITICAL: Connection refused [17:19:09] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:19:09] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:20:00] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [17:20:40] SSH on mayapple is CRITICAL: Server answer: [17:21:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [17:22:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:26:50] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [17:33:39] aliasd on yarrow is OK: TCP OK - 0.008 second response time on port 984 [500 Not found.] [17:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [17:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:37:50] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:37:50] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:38:19] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:38:19] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:38:19] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:38:39] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:39:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:39:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:39:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:39:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:39:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:39:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:39:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:40:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:40:41] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:42:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:43:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:44:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:45:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [17:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:46:20] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:47:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:47:41] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:48:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 871202.000000 [17:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:41] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:49:50] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1332355.000000 [17:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 636047.000000 [17:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:50:49] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:51:50] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 771940.000000 [17:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:52:19] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 409400.000000 [17:52:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:52:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:52:40] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:52:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:53:09] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:53:20] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2023852.000000 [17:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [17:53:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1224892.000000 [17:53:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:54:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54089 MB (8% inode=99%): [17:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:50] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:56:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:56:41] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:56:41] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:56:41] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [17:58:39] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 271056 MB (5% inode=62%): [17:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [18:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:02:09] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [18:06:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:06:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:10:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:19:11] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:19:59] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [18:20:40] SSH on mayapple is CRITICAL: Server answer: [18:21:40] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=92%): [18:23:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [18:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [18:33:10] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.165 second response time [18:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [18:36:20] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:37:50] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:39:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:39:19] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:39:19] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:39:39] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:39:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:40:10] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.729 second response time [18:40:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:40:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:41:09] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.435 second response time [18:41:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:45:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [18:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:47:00] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:47:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:19] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:39] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:48:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:49:10] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.358 second response time [18:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 871276.000000 [18:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:49:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1333398.000000 [18:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 635956.000000 [18:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:20] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 775541.000000 [18:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 408431.000000 [18:52:41] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:52:41] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:52:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:52:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:53:20] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2022497.000000 [18:53:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1226024.000000 [18:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [18:53:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:40] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3980 MB (13% inode=92%): [18:53:59] / on wolfsbane is WARNING: DISK WARNING - free space: / 3958 MB (13% inode=92%): [18:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53882 MB (8% inode=99%): [18:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:56:40] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [18:56:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:56:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:56:49] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:57:29] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [18:57:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [18:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 270851 MB (5% inode=62%): [18:59:09] / on ortelius is OK: DISK OK - free space: / 11281 MB (37% inode=92%): [18:59:40] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [18:59:40] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [18:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [19:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:03:30] MySQL on z-dat-s6-a is OK: Uptime: 768 Threads: 4 Questions: 4669 Slow queries: 1 Opens: 152 Flush tables: 1 Open tables: 106 Queries per second avg: 6.79 [19:03:40] MySQL slave on z-dat-s6-a is OK: Uptime: 773 Threads: 4 Questions: 5332 Slow queries: 1 Opens: 153 Flush tables: 1 Open tables: 107 Queries per second avg: 6.897 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 938 [19:06:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:06:09] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:10:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:13:40] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:13:40] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:17:40] /tmp on wolfsbane is OK: DISK OK - free space: / 6934 MB (23% inode=93%): [19:17:59] / on wolfsbane is OK: DISK OK - free space: / 6930 MB (23% inode=93%): [19:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:20:40] MySQL on z-dat-s3-a is OK: Uptime: 1616 Threads: 26 Questions: 667 Slow queries: 15 Opens: 151 Flush tables: 1 Open tables: 117 Queries per second avg: 0.412 [19:20:41] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2162 [19:20:41] SSH on mayapple is CRITICAL: Server answer: [19:23:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:26:52] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:40] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:28:40] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2418 [19:28:40] MySQL on z-dat-s7-a is OK: Uptime: 1274 Threads: 3 Questions: 1752 Slow queries: 1 Opens: 109 Flush tables: 1 Open tables: 96 Queries per second avg: 1.375 [19:31:40] MySQL slave on z-dat-s7-a is OK: Uptime: 1454 Threads: 15 Questions: 54585 Slow queries: 2 Opens: 478 Flush tables: 1 Open tables: 465 Queries per second avg: 37.541 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1689 [19:32:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [19:35:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:36:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:36:20] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:37:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:37:49] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:37:49] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:38:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:39:19] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:39:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:39:19] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:39:39] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:39:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:40:40] MySQL slave on z-dat-s3-a is OK: Uptime: 2818 Threads: 23 Questions: 343153 Slow queries: 54 Opens: 9107 Flush tables: 1 Open tables: 9068 Queries per second avg: 121.771 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1538 [19:40:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:40:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:41:39] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:47:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:47:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:21] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:48:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:48:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 874875.000000 [19:49:41] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:49:41] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:49:50] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1334275.000000 [19:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 639474.000000 [19:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:50:49] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:51:50] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:52:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 779140.000000 [19:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 406210.000000 [19:52:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:53:09] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:53:19] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2024042.000000 [19:53:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:53:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [19:53:41] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1227629.000000 [19:53:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:53:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:54:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53602 MB (8% inode=99%): [19:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:56:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:56:41] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:57:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:57:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:57:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:58:09] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 270503 MB (5% inode=62%): [19:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [20:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:00:49] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:19:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:21:42] SSH on mayapple is CRITICAL: Server answer: [20:23:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [20:36:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:36:20] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:37:50] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:39:19] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:39:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:39:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:39:39] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:39:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:40:00] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:40:40] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [20:40:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:40:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:41:41] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:47:00] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:47:40] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.020 second response time [20:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:19] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:48:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:48:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:48:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 873772.000000 [20:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1334775.000000 [20:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 638145.000000 [20:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:50:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:52:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 782741.000000 [20:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 403146.000000 [20:52:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:53:09] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:53:19] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2027023.000000 [20:53:39] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:53:39] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [20:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53367 MB (8% inode=99%): [20:54:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1229278.000000 [20:54:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:54:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:54:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:54:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:55:41] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:41] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:56:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:56:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:57:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:57:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:57:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:58:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:58:17] hello all [20:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 270181 MB (5% inode=62%): [20:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [21:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:03:23] 39? WTF? [21:04:14] ah, Debian 7 came out this weekend… [21:10:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:15:35] @notify legoktm [21:15:35] This user is now online in #wikimedia-dev so I will let you know when they show some activity (talk etc) [21:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:19:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:20:09] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:20:09] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:21:40] SSH on mayapple is CRITICAL: Server answer: [21:23:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [21:33:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:35:49] scfc_de: hi [21:35:59] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:36:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:37:49] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:37:49] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:39:03] DaB. * [Toolserver-announce] Upgrade to Debian 7 next Sunday [21:39:19] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:39:19] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:39:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:39:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:39:41] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:39:41] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:40] Load avg. on nightshade is WARNING: WARNING - load average: 24.34, 15.98, 9.27 [21:41:41] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:41:41] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:42:40] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:43:13] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:43:48] woooooo, wheezy [21:43:53] DaBPunkt: 2 questions... [21:44:04] * can we do something about this crazy level nagios spam [21:44:22] * how can i edit the description of [21:44:27] grr [21:44:29] Load avg. on nightshade is OK: OK - load average: 8.84, 14.32, 9.89 [21:44:44] * how can i edit the description of MNT-1322 (you just filed)? or only you can? [21:44:57] crazy level of nagios spam* [21:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:47:50] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:19] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:42] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:48:42] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:48:42] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:42] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 870971.000000 [21:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:49:41] jeremyb_: nosy works to make the switches accessable by nagios again – that will remove some of the nagios spam. You can ignore the bot if you like. [21:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1334981.000000 [21:50:31] AFAIK only the owner of a bug and the admins can edit the description. But I can add/change stuff if you like [21:50:39] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 636961.000000 [21:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:15] DaBPunkt: ok, but we could also e.g. limit broken checks to only alert once a day. or not at all. not critical but i think would be worth it to do. unless you think the switches will be fixed in less than say 48 hrs [21:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:51:49] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:04] DaBPunkt: i just wanted to add some space between sections (between new and removed for example). and maybe even seperate them into separate quotes. (break the blue line on the left side) [21:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 786341.000000 [21:52:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:52:19] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 399668.000000 [21:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:52:56] DaBPunkt: also, wondering what the latest status of the puppet repo is... can non-roots get access and submit patches? [21:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:53:19] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2030233.000000 [21:53:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:53:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [21:53:47] I have split the section as requested [21:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53105 MB (8% inode=99%): [21:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:54:50] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:54:50] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:55:38] jeremyb_: Operators can commit patches to puppt's SVN (in the trunk-section). They should also check and commit patches by non-operators. [21:55:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1230460.000000 [21:55:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:55:40] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:55:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:56:12] jeremyb_: about nagios: Do you have a quick how-to to do this? [21:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:40] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:56:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:57:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:57:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:57:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 269788 MB (5% inode=62%): [21:59:03] DaBPunkt: wooo, looks good [21:59:39] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [21:59:51] DaBPunkt: is there a web interface people could log in to? you could do it there. or by writing to a file. a lot depends on how the TS nagios is set up [22:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:00:44] our nagios webinterface is read-only [22:01:03] yeah, so then you'd have to do it by writing to a file :( [22:01:12] maybe i'll make a script for you to run [22:01:34] (not right now) [22:02:01] no stress. I have to leave for bed son anyway :-) [22:02:05] +o [22:02:10] i figured [22:10:09] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:10:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:19:09] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:19:09] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:20:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:20:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:20:26] DaBPunkt: so what's live and trunk? live is what's deployed right now and trunk is non-roots can commit? and deployment from svn to puppetmaster's manual? [22:20:36] * jeremyb_ just finished doing a git-svn clone of that repo [22:21:40] SSH on mayapple is CRITICAL: Server answer: [22:22:51] branches/live SHOULD be what is in puppet. trunk is where operators can commit. For unknow reasons there is also a "trunk"-folder in "live". [22:23:12] hah [22:23:38] so, why does e.g. puppet/modules/loginserver/manifests/motd.pp only exist in live but not trunk? [22:23:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:24:45] jeremyb_: because I have split before I noticed that some files where missing. [22:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [22:34:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:35:19] anyway, nacht [22:35:37] gn8 jeremyb_ [22:36:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:36:14] nacht ts [22:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:36:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:37:50] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:39:20] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:39:20] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:39:20] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:39:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:39:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:41:49] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:42:39] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:42:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:43:10] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:46:20] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:50] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:47:10] Free Memory on damiana is WARNING: WARNING - 6.2% (517560 kB) free! [22:47:19] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:48:20] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:48:20] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:48:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:48:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:48:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:09] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:09] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:19] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:39] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 869230.000000 [22:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:49:41] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:49:50] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1335301.000000 [22:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 636353.000000 [22:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:50:49] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:19] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:51:50] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 789941.000000 [22:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:52:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 396103.000000 [22:52:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:52:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:53:09] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:53:19] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2033588.000000 [22:53:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [22:53:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:54:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52853 MB (8% inode=99%): [22:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:55:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1232287.000000 [22:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:19] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:56:41] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:56:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:56:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:57:20] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:57:40] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:57:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:57:40] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:58:19] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:58:39] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 269493 MB (5% inode=62%): [22:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [23:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:10:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:16:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:19:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:19:10] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:19:11] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:19:40] aliasd on yarrow is CRITICAL: Connection refused [23:20:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:20:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:21:40] SSH on mayapple is CRITICAL: Server answer: [23:23:39] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:26:49] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:10] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (366844 kB) free! [23:29:10] Free Memory on damiana is WARNING: WARNING - 5.9% (494484 kB) free! [23:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [23:34:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:36:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:36:20] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:36:30] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:37:51] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:38:20] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:39:19] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:39:19] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:39:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:39:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:39:41] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:39:41] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:42:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:42:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:42:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:43:09] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:44:10] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (372392 kB) free! [23:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:00] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:47:49] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:19] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:48:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:48:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:40] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [23:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 867175.000000 [23:49:40] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [23:49:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [23:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:49:41] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [23:50:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 635218.000000 [23:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:50:41] RAID on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:50] SSH on rosemary is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:20] SMTP on rosemary is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:20] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [23:52:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:52:20] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 393191.000000 [23:52:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:53:19] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2036686.000000 [23:53:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:53:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [23:54:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:54:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52606 MB (8% inode=99%): [23:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:54:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:54:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:55:10] SMTP on rosemary is OK: SMTP OK - 0.002 sec. response time [23:55:10] RAID on rosemary is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:55:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1233353.000000 [23:55:41] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:41] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:41] SSH on rosemary is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:56:41] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:56:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:56:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:57:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:57:41] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:57:41] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:57:41] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 265882 MB (4% inode=62%): [23:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates).