[03:32:53] Hello! I've got an issue: how can I access remotely my database of user? [09:03:04] DaB. * Re: [Toolserver-l] SGE job queue availability and execution [09:20:05] hello all [14:37:28] @replag [14:37:30] scfc_de: s1-rr-a-wd: 1w 3d 9h 25m 44s [-]; s1-user-c: 6d 19h 11m 11s [-]; s1-user-wd: 2w 10h 23m 0s [-]; s2-rr: 3h 2m 35s [-]; s2-user: 3h 2m 35s [-]; s2-user-c: error; s2-user-wd: 1w 5d 22h 35m 26s [-]; s3-user: 60s [-] [14:37:31] scfc_de: s3-user-wd: 1w 3d 16h 25m 40s [-]; s4-user-wd: 3w 16h 50m 1s [-]; s5-rr-a-wd: 5d 22h 53m 51s [-]; s5-user: 1h 17m 45s [-]; s6-user-wd: 1w 3d 20h 41m 55s [-]; s7-user-wd: 1w 1d 17h 29m 41s [-] [14:43:42] DaBPunkt: Hi! Is the definition of ts_rc_age() or ts_replag.ts_replag available somewhere? [15:05:45] let me think [15:06:38] no, it is not afaik (ts_replag is just a view) [15:07:17] What's its definition? [15:07:30] CREATE OR REPLACE VIEW ts_replag AS SELECT ts_rc_age() AS ts_replag; [15:08:04] No, I meant ts_rc_age(). [15:08:47] scfc_de: http://pastebin.com/TmbhDbuz [15:10:09] DaBPunkt: Thanks! [15:10:13] np [18:10:23] DaBPunkt: Mail seems to be stuck on yarrow again. [18:15:12] Susan: are you the edwardsbot controler? [18:15:19] *operator [18:26:02] mayapple seems to be down. [18:26:19] (ssh-wise.) [18:28:38] scfc_de: yes. And I problems to find the managment-console for a reboot [18:29:28] +have [18:32:01] DaBPunkt: No problem. Could you flush the mail queue on yarrow as well? [18:32:11] yes, wait a moment [18:33:03] It looks like the alias-daemon is crashing again. [18:33:24] I guess we have finally get rid of it… [19:02:42] hello [19:02:45] maintenance tonight [19:02:50] \o/ [19:02:57] :D [19:03:11] * valhallasw hands nosy the WD-40 and duct tape [19:03:44] thanks [19:04:14] lets see if that holds until end of 2014... [19:04:34] no today only a little db maintenance - nothing hard [19:08:31] going to stop s5 on cassia [19:30:54] ok will now do the restarts [19:30:59] lets start with s3 [19:35:50] s4 is next [19:37:43] s6 now [19:40:48] ok s4 and s6 are back, s3 is still recovering [19:44:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1132221.000000 [19:44:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 940491.000000 [19:44:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 755255.000000 [19:44:40] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1249710.000000 [19:44:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:44:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:44:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [19:44:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63043 MB (10% inode=99%): [19:44:45] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:44:47] / on thyme is UNKNOWN: NRPE: Unable to read output [19:44:47] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:48] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:48] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:56] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:44:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:57] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:04] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:53] yea wb tsnag... [19:47:35] hmm, should qstat show jobs on all servers? [19:47:55] / on wolfsbane is OK: DISK OK - free space: / 6768 MB (22% inode=93%): [19:48:05] /tmp on wolfsbane is OK: DISK OK - free space: / 6756 MB (22% inode=93%): [19:49:59] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [19:50:04] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [19:50:04] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1876 [19:50:54] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:51:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:52:04] MySQL slave on z-dat-s2-b is OK: Uptime: 169155 Threads: 3 Questions: 154333442 Slow queries: 2706 Opens: 1749861 Flush tables: 1 Open tables: 256 Queries per second avg: 912.378 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1761 [20:00:24] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3085.000000 [20:00:34] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3070.000000 [20:00:44] MySQL on cassia is OK: Uptime: 45 Threads: 1 Questions: 7412 Slow queries: 0 Opens: 62 Flush tables: 1 Open tables: 55 Queries per second avg: 164.711 [20:00:44] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3037 [20:01:04] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2096 [20:01:24] ok s5 is back [20:01:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:01:50] s3 is close to coming back [20:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:02:55] MySQL on z-dat-s3-a is OK: Uptime: 1795 Threads: 22 Questions: 81 Slow queries: 2 Opens: 22 Flush tables: 1 Open tables: 15 Queries per second avg: 0.45 [20:03:35] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1717.000000 [20:03:45] MySQL slave on cassia is OK: Uptime: 226 Threads: 2 Questions: 166241 Slow queries: 2 Opens: 78 Flush tables: 1 Open tables: 71 Queries per second avg: 735.579 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1528 [20:04:04] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [20:04:04] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1909 [20:08:37] ok s3 is back too [20:08:44] maintenance done :) [20:08:47] *wave* [20:17:24] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1723.000000 [20:20:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:37:04] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2079 [20:43:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:43:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 296570 MB (5% inode=62%): [20:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:43:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:43:56] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:43:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:43:57] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:43:57] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:04] Load avg. on mayapple is CRITICAL: CRITICAL - load average: 2265.74, 2260.66, 2247.21 [20:44:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:44:05] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:05] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:05] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:05] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:05] MySQL slave on z-dat-s3-a is OK: Uptime: 4262 Threads: 21 Questions: 1928916 Slow queries: 168 Opens: 21299 Flush tables: 1 Open tables: 13375 Queries per second avg: 452.584 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1738 [20:44:06] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:06] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:44:15] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:44:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:25] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 609484.000000 [20:44:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1135389.000000 [20:44:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 940526.000000 [20:44:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 753805.000000 [20:44:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:36] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:44:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1248847.000000 [20:44:37] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:44:38] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:44:52] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:04] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:45:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:45:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:15] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1895812.000000 [20:45:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:45:16] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:01:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:04:05] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:18:04] Lars Aronsson * Re: [Toolserver-l] Cronie error [21:20:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:04] MySQL slave on z-dat-s2-b is OK: Uptime: 175155 Threads: 8 Questions: 162035220 Slow queries: 2831 Opens: 1804196 Flush tables: 1 Open tables: 256 Queries per second avg: 925.96 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1793 [21:40:35] Alchimista: Yes. [21:41:09] Susan: he is loving to work today: http://pt.wikipedia.org/wiki/Usu%C3%A1rio%28a%29_Discuss%C3%A3o:Alchimista#Forced_user_renames_coming_soon_for_SUL [21:41:10] :P [21:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 296141 MB (5% inode=62%): [21:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:43:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:43:55] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:14] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:14] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:44:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 613083.000000 [21:44:34] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1138352.000000 [21:44:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 938935.000000 [21:44:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 752103.000000 [21:44:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:36] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:36] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:37] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:37] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:44:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1247722.000000 [21:44:38] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:39] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:44:55] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:44:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:56] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:04] Load avg. on mayapple is CRITICAL: CRITICAL - load average: 2342.69, 2337.83, 2324.38 [21:45:05] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:45:05] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:05] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:05] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:06] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:06] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:07] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:45:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:16] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1899211.000000 [21:45:44] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:54:06] Yeah. [21:54:24] Well, Nemo seems to have taken the "hit the print button over and over again" approach to solving the problem. [21:54:43] That didn't seem to help. [21:56:30] It seems to be exposing a MediaWiki bug. [21:56:36] The code is very simple and stupid. [22:01:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:01:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:04:05] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [22:06:09] nacht ts [22:20:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 295696 MB (5% inode=62%): [22:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:43:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:43:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:44:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:25] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616684.000000 [22:44:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 515624.000000 [22:44:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62259 MB (10% inode=99%): [22:44:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:44:45] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:44:45] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:45] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:46] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:46] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:44:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:44:47] / on thyme is UNKNOWN: NRPE: Unable to read output [22:44:48] aliasd on mayapple is CRITICAL: No data received from host [22:45:04] Load avg. on mayapple is CRITICAL: CRITICAL - load average: 2418.65, 2413.59, 2399.95 [22:45:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:45:05] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:05] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:05] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:45:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:05] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:06] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:45:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:15] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:15] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:45:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1902716.000000 [22:45:34] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 749547.000000 [22:45:34] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1140218.000000 [22:45:34] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:45:34] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:34] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 939550.000000 [22:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:36] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:36] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:45:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1246995.000000 [22:45:37] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:38] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:38] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:39] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:01:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:01:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:04:04] APT on mayapple is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [23:20:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:15] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [23:42:04] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [23:43:55] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:43:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294835 MB (5% inode=62%): [23:43:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:43:55] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:43:56] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:43:56] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:44:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:15] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:15] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:44:24] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:24] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:25] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 620284.000000 [23:44:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:44:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62038 MB (10% inode=99%): [23:44:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512234.000000 [23:44:47] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:47] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:47] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:47] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:44:47] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [23:44:47] aliasd on mayapple is CRITICAL: No data received from host [23:44:48] / on thyme is UNKNOWN: NRPE: Unable to read output [23:44:48] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:44:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:44:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:55] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:55] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:55] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1905941.000000 [23:45:35] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:35] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:35] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:36] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 748266.000000 [23:45:36] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1141420.000000 [23:45:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 938712.000000 [23:45:37] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:45:38] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:38] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:39] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1246410.000000 [23:45:39] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:55] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:55] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:45:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:56] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:56] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:45:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:04] Load avg. on mayapple is CRITICAL: CRITICAL - load average: 2495.36, 2490.19, 2476.89 [23:46:04] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:46:04] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:46:04] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:46:04] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:46:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:05] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:06] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:46:06] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:46:14] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:46:14] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:46:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:14] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:15] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:46:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output