[00:01:33] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:03:33] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:45] / on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:06:53] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:06:53] Environment IPMI on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:07:04] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:10:33] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:14:22] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:01] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:02] SMTP on daphne is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:21] SSH on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:23] head and shoulders, knees and toes [00:33:26] thats what my daughter sings :D [00:34:38] nosy: in english? wow [00:34:42] yes :) [00:34:45] but not the baby [00:40:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:40:12] Authoritative DNS on ha-dns-auth is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:40:42] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:40:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76350.000000 [00:40:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77903 MB (12% inode=99%): [00:41:01] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131714.000000 [00:41:01] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:02] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:02] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:41:02] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:41:22] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:41:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [00:41:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:41:42] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:41:42] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:41:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 405286 MB (7% inode=64%): [00:41:52] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:42:02] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:42:02] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:42:11] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:42:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50943.000000 [00:42:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 173661.000000 [00:42:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 242686.000000 [00:42:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:42:31] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:42:31] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:42:31] / on thyme is UNKNOWN: NRPE: Unable to read output [00:42:41] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:42:42] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:45:05] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:25] Sensors on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:25] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:25] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:25] /tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:25] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:26] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:26] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:27] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:45] SMTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:04] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:04] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:04] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:04] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:24] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 408.02, 401.17, 383.01 [00:46:25] SRaid on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:25] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:25] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:26] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:46:35] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:35] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:35] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:35] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:35] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:35] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:36] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:36] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:37] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:37] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:38] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:39] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:55] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:55] / on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:55] Environment IPMI on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:46:55] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:47:04] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:47:35] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:45] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:45] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:46] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:46] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:47:46] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:46] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:46] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14421.000000 [00:48:45] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14422.000000 [00:48:45] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:45] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:45] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162385.000000 [00:49:25] nosy: a.ns and b.ns have an ipv6 record, but c.ns not? [00:49:25] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14459.000000 [00:49:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157363.000000 [00:49:34] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:34] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:34] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:35] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:50:23] speckmade: http://lists.wikimedia.org/pipermail/toolserver-l/2013-April/005852.html [00:50:28] Merlissimo: yes c is not in amsterdam [00:50:45] MySQL on ha-sql.esi is CRITICAL: Cant connect to MySQL server on ha-sql.esi (146) [00:50:47] Merlissimo: you mean i better stop maintenance soon? [00:50:58] :P [00:50:58] ? [00:53:45] MySQL on ha-sql.esi is OK: Uptime: 55 Threads: 2 Questions: 35 Slow queries: 0 Opens: 15 Flush tables: 1 Open tables: 8 Queries per second avg: 0.636 [00:55:25] SSH on willow is CRITICAL: Server answer: [00:55:25] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:56:25] SSH on z-dat-s2-b is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [01:01:34] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:03:34] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:56] toolserver is down? [01:07:00] or dns issue [01:07:25] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:07:40] ts is in maintenance [01:07:45] / on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:07:45] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:07:54] Environment IPMI on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:10:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31135 MB (3% inode=99%): [01:10:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:14:25] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:04] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:20:04] SMTP on daphne is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:25] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge/GE/bin/sol-amd64/qstat [01:35:25] toolserver.org HTTP on ortelius is CRITICAL: Connection refused [01:35:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:54] toolserver.org HTTP on wolfsbane is CRITICAL: Connection refused [01:37:55] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge/GE/bin/sol-amd64/qstat [01:38:04] oh my god [01:38:04] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:08] STOP tsnag @_@ [01:38:37] we know, shut up [01:38:37] :P [01:40:35] Authoritative DNS on ha-dns-auth is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:40:45] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:40:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79950.000000 [01:40:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77903 MB (12% inode=99%): [01:40:54] toolserver, y u no work? [01:40:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:41:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:41:04] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:41:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:41:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135335.000000 [01:41:26] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:41:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:41:45] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:41:46] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:41:46] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 405286 MB (7% inode=64%): [01:42:25] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:42:25] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:42:25] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:42:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54556.000000 [01:42:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177264.000000 [01:42:26] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:42:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 246288.000000 [01:42:27] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:42:35] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:35] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:42:35] / on thyme is UNKNOWN: NRPE: Unable to read output [01:42:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [01:42:54] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:43:04] Hey guys, seems like my quesiton didn't go through due to netsplit [01:43:05] I need help logging in [01:43:05] anyone has a minute? [01:43:35] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:43:35] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:45:04] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:26] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:26] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:26] Sensors on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:26] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:26] /tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:26] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:27] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:27] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:45] SMTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:04] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:05] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:05] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:05] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:25] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 515.95, 508.90, 490.66 [01:46:25] SRaid on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:26] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:26] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:26] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:46:35] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:35] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:35] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:36] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:36] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:36] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:37] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:37] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:38] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:39] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:39] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:55] / on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:55] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:55] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:46:55] Environment IPMI on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:05] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:47:35] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:45] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:45] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:45] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:45] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:47:45] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:46] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:46] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:11] * Help_Please needs help. Feel free to msg him [01:48:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18021.000000 [01:48:45] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18022.000000 [01:48:45] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:45] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:45] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:49:25] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18059.000000 [01:53:26] huji what is wrong? I'm not an admin but maybe I can help. [01:54:20] Many things, Magog_the_Ogre! [01:54:24] Help_Please <_< [01:54:29] My account in JIRA is LOST! It doesn't recognize me anymore [01:54:51] My password on wiki.ts didn't work, I asked for password reset but haven't gotten an email [01:54:56] it's like 15 minutes already [01:55:10] and I cannot log into TS, and I don't know how to reset my public key [01:55:24] any help with any of those is appreciated [01:55:26] SSH on willow is CRITICAL: Server answer: [01:55:26] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [01:56:18] right now the Toolserver is undergoing maintenance [01:56:33] that might have affected some of the problems you're seeing [01:57:15] hmm, good to now [01:57:17] know [01:57:30] I'll try again tomorrow then [01:57:45] thanks anyways [01:58:01] ttyl [01:58:27] later huji (too late, for sure) [02:00:43] I hate to be a pest, but I did not see this maintenance in the newsletter [02:00:47] when is it expected to be lifted? [02:01:35] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:03:59] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:02] Authoritative DNS on ha-dns-auth is OK: DNS OK: 0.055 seconds response time. 1.www.toolserver.org returns 91.198.174.203 [02:08:09] / on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:08:17] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:08:17] Environment IPMI on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:08:37] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:10:48] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:37:44] Looks like the web server is still having issues (can't get a connect on neither https or http) [03:34:36] Load avg. on yarrow is OK: OK - load average: 1.00, 1.00, 2.59 [03:34:36] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 183989.000000 [03:34:36] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24364.000000 [03:34:36] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:34:36] SMF on amaranth is OK: OK - all services online [03:34:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/17:DOWN:1 UP: OK [03:34:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/5:DOWN:1 UP: OK [03:34:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 253015.000000 [03:34:38] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:34:39] SMTP on amaranth is OK: SMTP OK - 0.488 sec. response time [03:34:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86785.000000 [03:34:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77705 MB (12% inode=99%): [03:34:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/0:UP:1 UP: OK [03:34:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/6:UP:1 UP: OK [03:34:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/18:DOWN:1 UP: OK [03:34:42] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:35:02] Sun Grid Engine execd on mayapple is UNKNOWN: Execution timeout exceeded [03:35:11] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:39:11] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [05:10:02] webserver is dead? [05:10:31] i'm also having problems with ssh to willow/nightshade so maybe it is more general problem [05:30:39] 06 05:10:02 < closedmouth> webserver is dead? [05:30:39] 06 05:10:31 < eranroz> i'm also having problems with ssh to willow/nightshade so maybe it is more general problem [05:30:42] nadar: ^ [05:30:47] (UTC) [05:30:50] k, ty [05:31:32] nosy: ^ [05:31:33] since not being familiar with wikimedia* i just thought I#d post it just in case [05:31:41] yeah, danke [05:41:08] toolserver is refusing my ssh key... [05:41:16] any clue? [05:43:50] mrdrini: there's several other problems reported (i.e. web services not working) [05:44:02] first report in here 30+ mins ago [05:44:30] are you the same as steward drini? [06:54:33] Is toolserver down? I can't login [06:55:09] i also can't ssh to willow/nightshade for few hours [07:04:48] jeremyb: ex steward drini [07:04:54] I resigned many months ago [07:07:07] Is there something we can do? [07:08:14] I guess wait, amir [07:28:00] ok [07:28:51] best solution :) [07:28:57] oops [07:29:06] nick coet|login [07:39:54] er... willow is not letting me in via ssh. It's asking for a password now (why? It has always asked for the ssh key password) and then does not accept a (valid) password [07:41:13] alkamid: you must wait i think I have the same problem no [07:41:15] w [07:42:32] have there been any changes recently? My cronie tasks stopped working properly. Something wrong with the paths to /bin/python2.7 (I'll paste the exact error when I can access the logs) [07:43:39] there's an issue with the toolserver atm [08:02:16] Hello. Is there any root present? I cannot login since yesterday. [08:05:48] we know :/ [08:05:51] yes [08:06:04] general problem? [08:06:06] it basically tracks down to routing/dns-issue [08:06:25] the right little screw will do but choose out of 500 :D [08:07:12] just to be sure: Problem at my side or at ts? [08:07:27] at ts [08:07:38] you cant do anything [08:07:42] ok, i'll shut up. thx. [08:15:48] sorry [08:16:13] but sleepless night requires some humor [08:35:00] can you please try to log in via willow? [08:35:11] ah nightshade is online again too.... [08:35:18] * legoktm tries [08:35:37] willow is asking for a password [08:35:37] sorry, that was a stupid mistake but hard to find... [08:36:13] legoktm@nightshade:~$ echo ":D" [08:36:14] :D [08:36:32] nosy: cant get into willow though [08:42:05] fine [08:42:10] at least i can ping now [08:44:21] nightshade took some really long for the first login but seems now normal. [08:44:51] ok now it should work [08:45:02] forgot to restart solaris ldap client.... [08:45:28] great, i'm in :D [08:45:32] thanks nosy :) [08:54:19] Load avg. on yarrow is OK: OK - load average: 3.77, 3.80, 3.32 [08:54:19] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 203171.000000 [08:54:19] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43546.000000 [08:54:19] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 186450.000000 [08:54:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 191472.000000 [08:54:20] SMF on amaranth is OK: OK - all services online [08:54:20] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [08:54:21] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [08:54:22] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [08:54:22] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [08:54:23] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [08:54:23] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [08:54:24] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:54:35] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [08:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [08:55:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [08:58:33] ls /mnt/user-store/ - dumps not found... where can i find the dumps? [09:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:00:24] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:00:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:00:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:00:42] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:00:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:00:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:08:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [09:08:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:09:03] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:09:03] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:11:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:13:23] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [09:20:33] MySQL on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [09:21:03] SSH on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:33] MySQL on z-dat-s5-b is OK: Uptime: 74 Threads: 1 Questions: 6054 Slow queries: 0 Opens: 76 Flush tables: 1 Open tables: 68 Queries per second avg: 81.810 [09:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:33] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:33] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:33] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:34] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:43] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43303 [09:53:43] aliasd on nightshade is CRITICAL: Connection refused [09:53:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:43] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:44] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:44] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:53:45] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:45] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:53:46] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 164887.000000 [09:53:54] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:02] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:03] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:03] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:54:03] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [09:54:14] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 195075.000000 [09:54:14] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43213.000000 [09:54:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 190052.000000 [09:54:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 206774.000000 [09:54:14] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:54:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 275797.000000 [09:54:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109566.000000 [09:54:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:24] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:24] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75495 MB (12% inode=99%): [09:54:24] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:54:25] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:26] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:26] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:27] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 80997 [09:54:28] / on thyme is UNKNOWN: NRPE: Unable to read output [09:54:43] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39384 [09:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:53] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:54:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:53] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:55:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81021.000000 [09:55:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47201.000000 [09:55:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47203.000000 [09:55:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:04] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:04] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:05] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:06] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:13] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:13] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:13] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:23] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [09:55:33] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:55:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:00:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:00:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:00:42] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:00:52] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:00:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:01:23] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:08:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [10:08:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:03] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:03] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:21:03] SSH on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:05] SSH on z-dat-s5-b is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [10:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:33] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:33] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:33] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:34] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:43] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35260 [10:53:43] aliasd on nightshade is CRITICAL: Connection refused [10:53:43] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:43] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:44] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 168486.000000 [10:53:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:03] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:03] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:03] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:54:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:04] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [10:54:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 198674.000000 [10:54:13] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35175.000000 [10:54:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 193652.000000 [10:54:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 210374.000000 [10:54:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113165.000000 [10:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:33] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:54:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 404486 MB (7% inode=64%): [10:54:43] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25594 [10:54:43] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:43] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:43] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:53] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:54:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:54] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:55:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83009.000000 [10:55:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50801.000000 [10:55:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50802.000000 [10:55:04] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:04] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:05] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:05] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:13] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:13] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:13] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:14] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:55:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 279457.000000 [10:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:23] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:23] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75231 MB (12% inode=99%): [10:55:24] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:55:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:25] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:26] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:26] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:55:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 83024 [10:55:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [10:56:03] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:00:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:00:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:00:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:00:46] I think API and bots are not working from willow... nosy ? [11:00:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:00:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:01:23] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:08:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [11:08:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:09:03] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:09:03] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:49:03] Is anyone else having problems with API and ssh? [11:50:31] jem-: its taking forever to login to willow [11:51:47] I'm inside from tonight, but API isn't working [11:52:11] At least I can see I'm not the only one [11:52:49] nosy: Around ? [11:53:08] and half my bots cannot connect to freenode [11:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:33] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:33] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:33] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:42] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15388 [11:53:43] aliasd on nightshade is CRITICAL: Connection refused [11:53:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:43] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:43] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 172086.000000 [11:53:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:02] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:02] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:02] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:54:03] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:04] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [11:54:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 202274.000000 [11:54:13] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15180.000000 [11:54:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 197251.000000 [11:54:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 213974.000000 [11:54:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116765.000000 [11:54:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:33] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:54:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:33] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 404136 MB (7% inode=64%): [11:54:43] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4926 [11:54:43] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:43] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:44] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:54:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:53] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:54:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:53] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:55:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84354.000000 [11:55:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54401.000000 [11:55:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54403.000000 [11:55:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:04] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:13] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:13] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:13] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:23] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:23] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74954 MB (12% inode=99%): [11:55:24] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:55:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:55:34] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:55:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:56:03] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:56:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:03] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:56:04] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:56:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:56:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 283117.000000 [11:56:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:56:23] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:56:23] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:56:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [11:56:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:56:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84324 [11:56:24] / on thyme is UNKNOWN: NRPE: Unable to read output [12:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:00:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:00:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:00:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:00:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3515 [12:00:52] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:00:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:00:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:01:23] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:08:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [12:08:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:03] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:03] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:14:38] @replag [12:31:43] MySQL slave on z-dat-s2-b is OK: Uptime: 12063 Threads: 11 Questions: 21168931 Slow queries: 171 Opens: 282457 Flush tables: 1 Open tables: 256 Queries per second avg: 1754.864 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1800 [12:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:33] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:33] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:34] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:42] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9333 [12:53:43] aliasd on nightshade is CRITICAL: Connection refused [12:53:43] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:43] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:44] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:44] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 174868.000000 [12:53:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:03] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:03] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:03] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [12:54:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:04] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [12:54:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 204924.000000 [12:54:13] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9282.000000 [12:54:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 199970.000000 [12:54:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 216084.000000 [12:54:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117490.000000 [12:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:33] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:54:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:33] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 403803 MB (7% inode=64%): [12:54:43] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:43] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:43] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:54:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:53] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:54:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:53] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:55:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84676.000000 [12:55:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53015.000000 [12:55:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50084.000000 [12:55:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:04] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:13] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:13] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:13] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:23] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:24] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74733 MB (12% inode=99%): [12:55:24] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:55:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:55:36] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:55:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:56:03] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:56:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:03] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:56:03] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:56:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:56:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 285744.000000 [12:56:23] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:56:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:56:23] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:56:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [12:56:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:56:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84666 [12:57:23] / on thyme is UNKNOWN: NRPE: Unable to read output [13:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:00:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:00:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:00:42] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:00:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:00:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:01:23] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:06:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2151 [13:07:59] @replag [13:08:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [13:08:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:03] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:03] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:24] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 28881 MB (2% inode=99%): [13:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:19:31] Hi all, why the codes run very very slow on toolserver ? [13:35:43] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3604 [13:39:43] Hi all, why the codes run very very slow on toolserver ? [13:40:27] Susan: reba seems to be having issues... [13:40:43] When I run a code it takes 4-5 mins for updating a page only [13:41:36] I saw a lot of complaints about toolserver tools not working since yesterday. [13:45:13] I'm not getting the API working at all, and of course my bot isn't editing [13:45:29] I'm beginning to feel a little worried [13:46:06] At least some information about the problem would help [13:47:19] my bot is editing fine through SGE on a linux host [13:47:47] yarrow to be specific [13:48:32] I had an SGE job killed this morning by the way. [13:51:07] legoktm: The Toolserver seems to be having issues. [13:51:30] I think its willow that's having issues... [13:51:40] nightshade/yarrow are running fine [13:52:03] I'm pretty sure my crontab's on willow. [13:52:15] I wish I finished migrating my tools to Wikitech :P [13:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:33] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:33] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:33] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:43] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8450 [13:53:43] aliasd on nightshade is CRITICAL: Connection refused [13:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:43] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:44] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 176938.000000 [13:53:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:03] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [13:54:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:03] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:03] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:04] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:13] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8462.000000 [13:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [13:54:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 206941.000000 [13:54:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 201993.000000 [13:54:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 219042.000000 [13:54:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116213.000000 [13:54:22] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:32] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:54:33] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 403500 MB (7% inode=64%): [13:54:43] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:43] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:43] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:43] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:53] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:54:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:54] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85476.000000 [13:55:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:55:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52235.000000 [13:55:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48989.000000 [13:55:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:04] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:13] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:13] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:13] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:23] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:23] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74601 MB (12% inode=99%): [13:55:24] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:55:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:55:33] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:55:52] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:56:02] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:56:02] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:03] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:56:03] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:56:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:56:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 287367.000000 [13:56:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:56:24] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:56:24] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:56:24] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [13:56:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:56:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85523 [13:57:23] / on thyme is UNKNOWN: NRPE: Unable to read output [14:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:00:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:00:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:00:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:00:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:00:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:01:24] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:03:49] Im still unable to get my IRC bots up [14:05:30] nosy1: Did you kill ipv6? [14:06:16] Darkdadaah: Sorry I missed you yesterday, you managed to ping me during dinner and I came back ~10m after you left. :-( [14:06:25] * Betacommand dropped email to ts mailing list [14:06:51] Coren: No problem. You have your life :-) [14:07:00] Betacommand: If I force ipv4 I can actually connect [14:07:19] multichill: I have no clue how to do that [14:07:42] How do you tell your bots to connect to what server? [14:08:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [14:08:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:09:03] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:09:03] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:15] John * [Toolserver-l] Issues [14:09:21] fsockopen($irc[0]['server'], $irc[0]['port']); using hostname [14:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:34] Betacommand: multichill@willow:~$ ping chat.freenode.net -> no answer from chat.freenode.net [14:09:50] multichill@willow:~$ ping -A inet chat.freenode.net -> chat.freenode.net is alive [14:10:02] * Betacommand uses irc.freenode.net [14:10:17] irc.freenode.net is an alias for chat.freenode.net. [14:10:33] ah [14:10:41] So your bots will try to use ipv6, that's broken, so your bots can't connect [14:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:12:45] however it looks like my python based bots are working, just not my php ones [14:13:31] only willow or for all servers? [14:13:49] Merlissimo: both willow and nightshade [14:14:14] Maarten Dammers * [Toolserver-l] Ipv6 issues [14:14:31] mayapple and nightshade work for me [14:14:38] nightshade ipv6 looks ok [14:15:49] inet6 addr: 2a02:ec80:101::2:3/70 <- wtf, a /70? [14:16:09] sge on willow is currently down, so all bots are using the other servers [14:16:26] Merlissimo: http://pastebin.com/f89y4fRw [14:17:02] Betacommand: on willow? [14:17:13] Merlissimo: nightshade [14:18:39] since willow is having issues i thought moving to nightshade was a better idea [14:18:40] ipv6 in terface is much faster for me than ipv4 [14:19:01] Betacommand: sge should do that for you [14:19:26] Merlissimo: Im not using sge [14:29:23] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36797 MB (6% inode=99%): [14:35:22] Darkdadaah: s/wikitech/labs/ [14:35:42] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5028 [14:36:35] jeremyb_: I thought it was merged? [14:36:58] Well, the sites at least. [14:38:55] Question: Hippietrail's account has been disabled. He has asked for it to be reinstated (it runs several tools for en.Wikt project). Is there any ETA for this reinstatement? [14:42:04] Amgine: Is there a ticket in JIRA? [14:43:27] Ah... Why did I not ask this? I've been aware of this for some time now :/ [14:44:18] Susan: https://jira.toolserver.org/browse/TS-1629 [14:45:10] Yes. I knew that... [14:46:15] Then e-mail ts-admins@toolserver.org with a link to that bug [14:46:15] . [14:46:22] If you get no response in a day or two, e-mail the mailing list. [14:46:27] toolserver-l@lists.wikimedia.org [14:51:02] Done [14:52:26] Cool. [14:52:30] Let me know if you don't hear back from anyone. [14:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:32] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:33] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:33] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:42] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7503 [14:53:43] aliasd on nightshade is CRITICAL: Connection refused [14:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:43] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:44] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 176373.000000 [14:53:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:03] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [14:54:03] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:03] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:03] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:04] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [14:54:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208956.000000 [14:54:13] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7479.000000 [14:54:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 204058.000000 [14:54:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 220623.000000 [14:54:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115556.000000 [14:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:33] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:54:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:33] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 403015 MB (7% inode=64%): [14:54:43] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:43] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:43] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:53] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:54:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:53] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:55:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86577.000000 [14:55:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:55:02] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53174.000000 [14:55:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36926.000000 [14:55:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:12] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:12] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:12] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:13] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:22] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:23] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:55:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74481 MB (12% inode=99%): [14:55:33] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:55:48] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3596 [14:55:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:03] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:03] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:03] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:56:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 288885.000000 [14:56:23] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:56:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:23] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:24] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [14:56:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:56:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86577 [14:57:23] / on thyme is UNKNOWN: NRPE: Unable to read output [15:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:00:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:00:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:00:42] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:00:52] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:00:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:01:23] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:08:58] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [15:08:58] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:09:03] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:09:03] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:32:43] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3584 [15:33:13] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3498.000000 [15:45:43] MySQL slave on z-dat-s5-b is OK: Uptime: 23125 Threads: 6 Questions: 24900170 Slow queries: 281 Opens: 1772 Flush tables: 1 Open tables: 256 Queries per second avg: 1076.764 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1642 [15:46:12] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1567.000000 [15:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:33] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:33] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:33] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:43] aliasd on nightshade is CRITICAL: Connection refused [15:53:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:43] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:43] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177289.000000 [15:53:58] @replag [15:54:03] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [15:54:03] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:03] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:03] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:04] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [15:54:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 210974.000000 [15:54:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 206194.000000 [15:54:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 222395.000000 [15:54:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114778.000000 [15:54:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:33] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:54:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 402601 MB (7% inode=64%): [15:54:33] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:42] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:43] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:43] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:43] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:54] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:54] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:54:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:54] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:55:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42407.000000 [15:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87297.000000 [15:55:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23274.000000 [15:55:04] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:04] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:55:13] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:13] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:14] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:23] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:23] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:55:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74358 MB (12% inode=99%): [15:55:33] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:55:43] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3295 [15:55:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:07] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:07] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:07] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:56:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 290237.000000 [15:56:23] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:56:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:24] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:24] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:24] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:24] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [15:56:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87339 [15:56:25] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:57:23] / on thyme is UNKNOWN: NRPE: Unable to read output [16:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:00:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:00:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:00:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:00:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:01:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:01:23] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:09:02] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:02] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:09:02] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:09:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [16:09:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:23:56] @replag [16:24:31] mzmcbride@willow:~$ ping google.com [16:24:31] no answer from google.com [16:24:33] :-( [16:24:36] The bot is dead? [16:25:07] Which bot? [16:25:11] Oh, tsbot. [16:25:12] And reba. [16:25:17] Yeah, some sort of network issue, I imagine. [16:28:31] Susan, there seems to be ipv6 problems [16:31:24] On a side note, I finally managed to completely copy one of my main tools to the Wikitech/Labs site. I was tempted to switch, but the tools seem to work ok on Toolserver now. [16:32:43] MySQL slave on z-dat-s2-b is OK: Uptime: 26523 Threads: 14 Questions: 44262551 Slow queries: 363 Opens: 472693 Flush tables: 1 Open tables: 256 Queries per second avg: 1668.836 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1682 [16:46:43] Load avg. on ptolemy is WARNING: WARNING - load average: 18.00, 16.88, 12.52 [16:50:43] Load avg. on ptolemy is OK: OK - load average: 6.84, 13.22, 12.16 [16:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:33] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:33] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:34] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:43] aliasd on nightshade is CRITICAL: Connection refused [16:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:43] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:43] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180224.000000 [16:54:03] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [16:54:03] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:03] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:03] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:04] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [16:54:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 213050.000000 [16:54:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208237.000000 [16:54:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 223076.000000 [16:54:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113684.000000 [16:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:32] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:54:42] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:44] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:44] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:44] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:53] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:54:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:54] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:55:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29723.000000 [16:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88698.000000 [16:55:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5234.000000 [16:55:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:55:13] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:13] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:13] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:23] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:23] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74245 MB (12% inode=99%): [16:55:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 402313 MB (7% inode=64%): [16:55:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:33] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:55:33] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:55:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:56:03] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:56:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:03] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:56:03] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:56:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:56:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 291302.000000 [16:56:22] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:56:23] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:56:23] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:56:23] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:56:23] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:57:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [16:57:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:57:23] / on thyme is UNKNOWN: NRPE: Unable to read output [16:57:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88755 [17:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:00:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:00:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:00:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:00:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:01:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:01:03] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3554.000000 [17:01:23] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:04:41] Hello [17:08:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1533.000000 [17:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:03] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:09:03] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:09:03] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:09:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [17:09:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:18:15] Hersfold Wikipedia * Re: [Toolserver-l] Issues [17:37:33] [[User talk:Nullzero]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7898&rcid=21744 * Nullzero * (+39) (Created page with "Contact me at [[:th:User:Nullzero]]. :)") [17:53:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:23] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:23] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:33] FC 0/23 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:33] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:43] eth0 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:43] aliasd on nightshade is CRITICAL: Connection refused [17:53:43] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:43] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:43] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:44] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:53:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181715.000000 [17:53:53] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:03] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [17:54:03] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:03] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:03] FC 0/6 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:03] FC 0/18 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:04] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [17:54:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 215117.000000 [17:54:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 210122.000000 [17:54:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 224634.000000 [17:54:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112734.000000 [17:54:23] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:23] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:32] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:54:43] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:54:43] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:43] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:43] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:44] FC 0/16 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:54:53] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:53] FC 0/17 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:53] FC 0/5 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:54:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13948.000000 [17:55:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90434.000000 [17:55:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:55:03] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:04] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:13] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:13] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:13] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:14] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:14] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:23] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:23] FC 0/20 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:23] FC 0/9 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:23] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74105 MB (12% inode=99%): [17:55:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 402066 MB (7% inode=64%): [17:55:43] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:43] FC 0/14 [hyacinth] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:55:53] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:55:53] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:56:03] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:56:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:03] FC 0/19 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:56:03] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:56:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:56:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 292826.000000 [17:56:24] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:56:24] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:56:24] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:56:24] FC 0/21 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:56:24] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:57:23] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [17:57:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:57:23] / on thyme is UNKNOWN: NRPE: Unable to read output [17:57:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90508 [17:59:03] Load avg. on daphne is WARNING: WARNING - load average: 17.46, 15.70, 11.11 [18:00:23] FC 0/22 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [18:00:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:00:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:00:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:00:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:00:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:00:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:01:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:01:23] FC 0/13 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [18:04:03] Load avg. on daphne is OK: OK - load average: 12.41, 14.39, 11.90 [18:09:03] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:09:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:09:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:09:23] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:23] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:09:53] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [18:09:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:11:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:20:28] what is nightshade's public IP address? [18:23:53] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:24:23] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35826 MB (6% inode=99%): [18:31:18] Betacommand: 185.15.59.201 / 2a02:ec80:101::2:3 according to 8.8.8.8 [18:32:03] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3578.000000 [18:32:43] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2198 [18:32:43] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:34:02] valhallasw: thanks [18:35:43] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:35:43] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [18:35:43] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [18:42:02] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1742.000000 [18:42:43] MySQL on z-dat-s6-a is OK: Uptime: 788 Threads: 11 Questions: 462 Slow queries: 16 Opens: 112 Flush tables: 1 Open tables: 86 Queries per second avg: 0.586 [18:42:43] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2089 [18:42:43] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2498 [18:49:50] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 226465.000000 [18:49:50] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:49:50] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 213283.000000 [18:49:50] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (null) [18:49:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [18:49:51] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [18:49:52] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [18:49:52] FC 0/2 on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [18:49:53] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [18:49:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 294564.000000 [18:49:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:49:54] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112436.000000 [18:55:03] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2762 [18:55:03] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:55:13] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3527 [18:55:13] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2538 [18:55:13] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:55:33] FC 0/0 [far1-n1-oe16-esams A2] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/0:UP:1 UP: OK [18:55:34] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/3:UP:1 UP: OK [18:55:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/15:UP:1 UP: OK [18:55:34] FC 0/6 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/6:DOWN:1 UP: OK [18:55:34] FC 0/18 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/18:DOWN:1 UP: OK [18:55:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/4:UP:1 UP: OK [18:55:34] FC 0/1 [far1-n1-oe16-esams B2] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/1:UP:1 UP: OK [18:55:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/16:DOWN:1 UP: OK [18:55:35] FC 0/19 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/19:DOWN:1 UP: OK [18:55:35] FC 0/7 [daphne] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/7:UP:1 UP: OK [18:55:43] FC 0/17 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/17:DOWN:1 UP: OK [18:55:43] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/10:UP:1 UP: OK [18:55:44] FC 0/5 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/5:DOWN:1 UP: OK [18:55:44] FC 0/8 [adenia] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/8:UP:1 UP: OK [18:55:44] FC 0/2 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/2:DOWN:1 UP: OK [18:55:53] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/0:UP:1 UP: OK [18:55:53] FC 0/9 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/9:DOWN:1 UP: OK [18:55:54] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/6:UP:1 UP: OK [18:55:54] FC 0/20 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/20:DOWN:1 UP: OK [18:55:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/18:DOWN:1 UP: OK [18:55:54] FC 0/11 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/11:UP:1 UP: OK [18:55:54] FC 0/12 [thyme] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/12:UP:1 UP: OK [18:55:54] FC 0/19 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/19:DOWN:1 UP: OK [18:55:54] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/1:UP:1 UP: OK [18:55:55] FC 0/21 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/21:DOWN:1 UP: OK [18:55:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/7:UP:1 UP: OK [18:55:56] FC 0/13 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/13:DOWN:1 UP: OK [18:56:13] FC 0/21 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/21:DOWN:1 UP: OK [18:56:13] FC 0/15 [fsw1-n1-oe16-esams] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/15:UP:1 UP: OK [18:56:13] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/12:UP:1 UP: OK [18:56:13] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/3:UP:1 UP: OK [18:56:13] FC 0/13 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/13:DOWN:1 UP: OK [18:56:14] FC 0/22 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/22:DOWN:1 UP: OK [18:56:14] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/4:UP:1 UP: OK [18:56:15] FC 0/16 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/16:DOWN:1 UP: OK [18:56:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/14:DOWN:1 UP: OK [18:56:24] FC 0/17 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/17:DOWN:1 UP: OK [18:56:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/23:DOWN:1 UP: OK [18:56:24] eth0 on fsw1-n1-oe16-esams.mgmt is OK: eth0:UP:1 UP: OK [18:56:24] FC 0/5 on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/5:DOWN:1 UP: OK [18:57:12] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3646 [19:00:59] Hi where I can run my Pywikipediabot scripts, willow is out, and yarrow has got Python 2.6? [19:13:03] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3616 [19:14:16] Kolossos * Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster [19:18:13] MySQL slave on z-dat-s6-a is OK: Uptime: 2918 Threads: 16 Questions: 269304 Slow queries: 389 Opens: 2231 Flush tables: 1 Open tables: 312 Queries per second avg: 92.290 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1478 [19:28:15] DaB. * Re: [Toolserver-l] Ipv6 issues [19:28:18] coet|cawiki: you pywikipediabot should run on all servers using sge. [19:28:23] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:28:44] what is sge? [19:29:34] our grid scheduling system which run job on servers which are available and have lowest load [19:30:03] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35823 MB (6% inode=99%): [19:30:13] currently no job are running on willow because of some problem with this server [19:30:26] interesting option, i'll read about it... [19:30:42] https://wiki.toolserver.org/view/Job_scheduling [19:31:25] ah ok I know it [19:31:55] my trouble was that on willow i use the "python2.7" order [19:32:19] i remove the 2.7 ending, and i think i'll run fine [19:32:36] *it'll [19:33:10] alls python version on ts have patched the unicode bug [19:37:44] ok thanks [19:41:23] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:44:15] Marlen Caemmerer * [Toolserver-l] Failed Maintenance - was - Re: IP Renumbering of the complete Toolserver cluster [19:48:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:49:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:49:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 400809 MB (7% inode=64%): [19:49:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:49:23] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [19:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180640.000000 [19:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 228769.000000 [19:49:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:49:44] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 295249.000000 [19:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112535.000000 [19:49:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:50:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 216912.000000 [19:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:50:13] aliasd on nightshade is CRITICAL: Connection refused [19:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:50:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:50:22] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [19:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93538.000000 [19:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:50:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:34] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73722 MB (12% inode=99%): [19:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:50:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:50:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 93541 [19:50:54] / on thyme is UNKNOWN: NRPE: Unable to read output [19:55:03] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:55:13] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:57:53] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7228 [20:00:52] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35823 MB (6% inode=99%): [20:08:03] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3492 [20:09:06] Hi everyone, can you have a look on this? [20:09:24] ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 [20:13:03] MySQL slave on z-dat-s4-a is OK: Uptime: 107597 Threads: 1 Questions: 8898788 Slow queries: 2 Opens: 90 Flush tables: 1 Open tables: 66 Queries per second avg: 82.704 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1347 [20:19:03] MySQL on z-dat-s7-a is OK: Uptime: 61 Threads: 3 Questions: 4462 Slow queries: 0 Opens: 264 Flush tables: 1 Open tables: 252 Queries per second avg: 73.147 [20:25:12] aliasd on nightshade is OK: TCP OK - 0.005 second response time on port 984 [500 Not found.] [20:26:23] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:30:03] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35496 MB (6% inode=99%): [20:35:23] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:48:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:49:23] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [20:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181634.000000 [20:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 231028.000000 [20:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112766.000000 [20:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 400383 MB (7% inode=64%): [20:50:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:50:23] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94147.000000 [20:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:50:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:50:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 294225.000000 [20:50:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 219255.000000 [20:50:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 224906.000000 [20:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73480 MB (12% inode=99%): [20:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:50:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:50:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 94150 [20:50:54] / on thyme is UNKNOWN: NRPE: Unable to read output [20:55:12] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5984 [20:58:13] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7070 [21:03:15] Marlen Caemmerer * Re: [Toolserver-l] Ipv6 issues [21:08:53] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36257 MB (6% inode=99%): [21:09:17] Merlissimo * Re: [Toolserver-l] Ipv6 issues [21:13:13] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3354 [21:21:13] MySQL slave on z-dat-s7-a is OK: Uptime: 3789 Threads: 6 Questions: 5146634 Slow queries: 202 Opens: 861 Flush tables: 1 Open tables: 840 Queries per second avg: 1358.309 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1586 [21:26:13] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3524 [21:34:13] MySQL slave on z-dat-s3-a is OK: Uptime: 112458 Threads: 16 Questions: 83215578 Slow queries: 26326 Opens: 488225 Flush tables: 1 Open tables: 16383 Queries per second avg: 739.970 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1744 [21:36:13] aliasd on nightshade is CRITICAL: Connection refused [21:48:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:49:23] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 183332.000000 [21:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 233518.000000 [21:49:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113241.000000 [21:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 399943 MB (7% inode=64%): [21:50:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:50:23] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94616.000000 [21:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:50:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:50:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 293451.000000 [21:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73244 MB (12% inode=99%): [21:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:51:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 227070.000000 [21:51:23] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 220821.000000 [21:51:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:51:53] / on thyme is UNKNOWN: NRPE: Unable to read output [21:51:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 94643 [22:12:13] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [22:44:54] http://toolserver.org/~magnus/catscan_rewrite.php?categories=All+free+media&ns%5B6%5D=1&templates_no=Information%0D%0Aprotected+generic+image+name&smaller=50&sortby=size&doit=1 [22:45:04] Why is this returning a number of entries? [22:45:08] Qcoder00: looking [22:45:10] It shouldn't be returning any [22:45:44] Hullo [22:45:55] Is this the right chan for db queries? [22:47:35] greenrosetta: depends on what you need [22:47:47] I'm interested in wrting a query [22:48:06] ok..... [22:48:08] I know my sql well enough [22:48:38] Toolserver has the ability to host a query I want to create? [22:48:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:49:16] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:16] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:16] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:16] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:49:21] greenrosetta: yes [22:49:23] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [22:49:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 184393.000000 [22:49:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:27] @replag [22:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 235708.000000 [22:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113235.000000 [22:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:49:59] Betacommand: I applied for an account already, but while I wait, is there a sample DB I can download to experiment now? [22:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 399689 MB (7% inode=64%): [22:50:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:50:21] greenrosetta: the databases are rather large [22:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:50:23] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:50:33] How about a list of tables? [22:50:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:50:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94080.000000 [22:50:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:50:33] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:50:44] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 293066.000000 [22:50:52] greenrosetta: http://www.mediawiki.org/wiki/Manual:Database_layout [22:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72890 MB (11% inode=99%): [22:50:54] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:54] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:51:00] Qcoder00: looks like lag issues [22:51:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 221937.000000 [22:51:48] Betacommand: maybe what I want to write already exists. I'd like to get a list of distinct users that have contributed to a list of articles [22:51:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 229103.000000 [22:51:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:51:53] / on thyme is UNKNOWN: NRPE: Unable to read output [22:51:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 93968 [22:52:33] Qcoder00: looks like lag issues [22:52:55] greenrosetta: its far better to break it into multiple queries [22:53:40] select distinct(rev_user_text) from revision where rev_page = ; [22:54:02] and then do an or operation? [22:54:07] I mean and [22:54:27] greenrosetta: just combine the lists [22:54:49] greenrosetta: dont combine the queries [22:54:51] how would that work on TS? does it use TSQL or something? [22:55:23] Ideally I'd like to be able to publish this query for everyone to use on TS [22:55:32] greenrosetta: it requires external processing [22:56:10] select distinct(rev_user_text) from revision where rev_page in (A,B,C); [22:56:11] greenrosetta: anyone who uses the toolserver knows basic quires like that [22:56:20] greenrosetta: dont do that [22:56:28] overheat the server? [22:56:38] greenrosetta: it will take forever [22:57:08] im not anticipating the list size to be very large [22:57:18] greenrosetta: it doesnt matter [22:57:32] page in () sucks for performance reasons [22:57:35] why, is rev_page not indexed well? [22:57:37] ah [22:58:00] I assume rev_page ="a" OR rev_page="B" is going to be just as bad? [22:58:06] yes [22:58:18] so how do people do the external processing? [22:58:49] greenrosetta: do you program at all? [22:58:57] yes [22:59:10] it sounds like you are saying [22:59:24] that I cant do what I want all on a toolserver account [22:59:34] get a list of users for each page, feed them into a scrip and sort, and remove duplicates [22:59:41] yes, thats simple enough [22:59:41] greenrosetta: its easily done [22:59:58] it cannot be done solely with sql queries [23:00:00] so im confused a bit how toolserver works [23:00:28] greenrosetta: the toolserver is a set of servers [23:00:36] ah [23:01:04] it is broken into several groups, SQL, login servers, web servers, osm, and a few that I cant recall off hand [23:01:23] so lets say they let me have an account [23:01:41] you get a /home directory [23:01:50] greenrosetta: what languages do you program in? [23:01:52] And I can create my ownphp page? [23:01:57] yes [23:02:05] Im a .NET guy, but I can figure most anything out [23:02:18] syntax is syntax [23:03:15] http://toolserver.org/~betacommand/cgi-bin/img_removal.py?title=The_Oates%27_Valor [23:03:26] greenrosetta: .net will NOT run on the toolserver [23:03:27] I could call the query from the php page and parse it using the webserver [23:03:32] yeah I know that [23:04:00] greenrosetta: if you want you can just write a php script and run in from the login servers [23:04:37] Do I telnet in? [23:04:48] SSH [23:05:02] greenrosetta: https://wiki.toolserver.org/view/Getting_started [23:05:09] good place to start [23:05:26] read that and related pages and then come back [23:06:07] one last question. Can toolserver host web-services? [23:07:18] define web services [23:07:46] I pass in (A,B,C) to a URL and it returns me (User1, User2, User3) [23:08:02] Ideally I'd write that and then have my php page just call it [23:08:27] greenrosetta: http://toolserver.org/~betacommand/cgi-bin/img_removal.py?title=The_Oates%27_Valor [23:08:33] something like that? [23:10:03] Yes, though Ideally I'd have a php page call that and then format the output how I want it [23:10:18] But the format isn't unusable so it would work just fine [23:10:40] what is that python? [23:10:50] yeah [23:11:29] So I could create a page with let's say 10 text boxes, then call a python script like yours, then display the results? [23:15:39] greenrosetta: you can do it in php if you want [23:16:09] makes no difference. I've never used python before, but I'm sure I can figure it out [23:19:18] who approves access to ts? [23:48:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:49:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:49:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:49:13] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:13] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:13] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:13] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:49:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 184873.000000 [23:49:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 237773.000000 [23:49:53] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113198.000000 [23:49:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:50:03] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:50:03] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:50:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 399606 MB (7% inode=64%): [23:50:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:50:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:50:13] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:50:22] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [23:50:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:50:23] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:50:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:50:32] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:50:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91485.000000 [23:50:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:50:32] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:46] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:46] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:50:46] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 292458.000000 [23:50:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72619 MB (11% inode=99%): [23:50:53] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:53] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:51:23] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 223306.000000 [23:51:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 230965.000000 [23:51:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [23:51:53] / on thyme is UNKNOWN: NRPE: Unable to read output [23:51:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91419