[00:03:25] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:16:44] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:16:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:17:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:15] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:15] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 311212.000000 [00:17:35] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65432 MB (10% inode=99%): [00:17:35] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:35] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:35] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:35] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 387220 MB (7% inode=63%): [00:17:36] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355234.000000 [00:17:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36516.000000 [00:17:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 253094.000000 [00:17:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 311636.000000 [00:17:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 171638.000000 [00:17:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:17:55] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16049.000000 [00:18:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:18:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:18:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:18:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:05] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:18:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:18:06] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:34] / on thyme is UNKNOWN: NRPE: Unable to read output [00:18:55] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:18:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:18:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 250964.000000 [00:19:05] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [00:19:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:19:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:19:35] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:19:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:28:37] nacht ts [00:47:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:03:25] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:11:55] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3499.000000 [01:15:55] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1689.000000 [01:16:45] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:16:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:17:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:15] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:15] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 312099.000000 [01:17:34] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:34] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:35] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:35] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64963 MB (10% inode=99%): [01:17:35] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 390106 MB (7% inode=63%): [01:17:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355704.000000 [01:17:44] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18312.000000 [01:17:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 252431.000000 [01:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 312778.000000 [01:17:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 168934.000000 [01:17:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:18:04] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:18:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:18:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:18:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:05] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:18:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:18:05] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:35] / on thyme is UNKNOWN: NRPE: Unable to read output [01:18:55] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:18:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:18:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 251912.000000 [01:19:05] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [01:19:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:19:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:19:34] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:34:55] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31828 MB (3% inode=99%): [01:47:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:57:45] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2826.000000 [01:59:45] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1557.000000 [02:03:25] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:16:45] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:16:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:17:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:15] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:15] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:24] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 313884.000000 [02:17:34] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:17:34] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:17:34] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:17:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64737 MB (10% inode=99%): [02:17:35] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 389267 MB (7% inode=63%): [02:17:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 356320.000000 [02:17:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 251053.000000 [02:17:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 314363.000000 [02:17:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 167247.000000 [02:17:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:18:04] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:18:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:18:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:18:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:05] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [02:18:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:18:05] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:35] / on thyme is UNKNOWN: NRPE: Unable to read output [02:18:55] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:18:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:18:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 251373.000000 [02:19:05] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [02:19:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:19:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:19:34] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:47:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:03:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:06:55] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (244548 kB) free! [03:10:54] Free Memory on damiana is WARNING: WARNING - 6.9% (576668 kB) free! [03:11:55] Free Memory on damiana is OK: OK - 8.1% (679868 kB) free. [03:16:45] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:16:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:17:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:17:15] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:17:15] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:17:24] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:17:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 315704.000000 [03:17:34] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:17:34] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:17:34] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:17:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64605 MB (10% inode=99%): [03:17:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 357082.000000 [03:17:35] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 389035 MB (7% inode=63%): [03:17:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 250048.000000 [03:17:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 316406.000000 [03:17:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 165247.000000 [03:17:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:18:05] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:18:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:18:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:18:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:06] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:18:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:18:06] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:35] / on thyme is UNKNOWN: NRPE: Unable to read output [03:18:55] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:18:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:18:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 253005.000000 [03:19:05] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [03:19:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:19:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:19:34] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:47:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:54:55] Free Memory on damiana is CRITICAL: CRITICAL - 2.1% (177788 kB) free! [03:59:55] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 33671 MB (6% inode=99%): [04:02:55] Free Memory on damiana is WARNING: WARNING - 5.7% (474840 kB) free! [04:03:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:03:55] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (345336 kB) free! [04:16:45] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:16:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:17:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:15] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:15] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:24] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 317641.000000 [04:17:34] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:34] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:34] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64410 MB (10% inode=99%): [04:17:35] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 389745 MB (7% inode=63%): [04:17:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 357925.000000 [04:17:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 249913.000000 [04:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 318963.000000 [04:17:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 163709.000000 [04:17:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:18:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:18:06] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:18:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:18:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:18:06] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:18:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:18:07] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:18:15] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:18:35] / on thyme is UNKNOWN: NRPE: Unable to read output [04:18:55] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:18:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:18:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 253285.000000 [04:19:05] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [04:19:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:19:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:19:35] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:19:35] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:47:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:03:34] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:48:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:49:09] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:18] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:59] Free Memory on damiana is OK: OK - 91.9% (7700952 kB) free. [05:53:50] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:53:59] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [05:54:20] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1953.000000 [05:54:20] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2957.000000 [05:54:20] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2958.000000 [05:54:29] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2959.000000 [05:54:40] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1946 [05:57:21] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1770.000000 [05:57:41] MySQL slave on rosemary is OK: Uptime: 1002801 Threads: 10 Questions: 892992510 Slow queries: 299708 Opens: 45709 Flush tables: 1 Open tables: 3390 Queries per second avg: 890.498 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1722 [06:05:20] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3617.000000 [06:05:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [06:05:30] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [06:12:00] Sun Grid Engine execd on mayapple is UNKNOWN: Execution timeout exceeded [06:12:41] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:59] SSH on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:19] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [06:30:50] SSH on yarrow is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [06:31:09] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [06:37:50] SSH on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:10] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [06:38:29] SSH on mayapple is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [06:38:39] Sun Grid Engine execd on mayapple is OK: Host and Queues Ok [06:47:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:47:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:47:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:47:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 389563 MB (7% inode=63%): [06:47:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:48:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:48:09] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [06:48:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 259228.000000 [06:48:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:48:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:48:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:48:20] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:48:20] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:48:20] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:48:20] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:48:21] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:21] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 325299.000000 [06:48:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 326708.000000 [06:48:30] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 255758.000000 [06:48:30] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:30] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:48:31] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 364819.000000 [06:48:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 169403.000000 [06:48:40] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:48:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63962 MB (10% inode=99%): [06:48:40] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:40] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:48:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:48:41] / on thyme is UNKNOWN: NRPE: Unable to read output [06:49:09] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:19] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:53:50] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:54:04] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:00:50] SSH on nightshade is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [07:00:50] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [07:05:29] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7219.000000 [07:06:20] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7277.000000 [07:06:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7278.000000 [07:18:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:48:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:48:09] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [07:48:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 262829.000000 [07:48:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:10] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:48:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:48:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:48:35] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 330308.000000 [07:48:35] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 328899.000000 [07:48:35] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 259358.000000 [07:48:35] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:35] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:48:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368419.000000 [07:48:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 173003.000000 [07:48:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63149 MB (10% inode=99%): [07:48:40] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:40] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:48:40] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:48:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:48:41] / on thyme is UNKNOWN: NRPE: Unable to read output [07:48:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:48:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:48:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:48:49] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 389355 MB (7% inode=63%): [07:48:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:49:09] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:49:19] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:49:19] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:19] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:20] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:20] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:21] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:21] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:50] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:53:59] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [08:06:29] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10879.000000 [08:07:19] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10937.000000 [08:07:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10938.000000 [08:18:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:48:09] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [08:48:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 266428.000000 [08:48:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:48:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:48:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 332499.000000 [08:48:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 262957.000000 [08:48:37] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 333908.000000 [08:48:37] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:37] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:48:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 372019.000000 [08:48:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:48:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:48:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:48:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 389082 MB (7% inode=63%): [08:48:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:48:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:49:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:49:20] Load avg. on nightshade is WARNING: WARNING - load average: 16.52, 15.45, 12.88 [08:49:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 176663.000000 [08:49:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62661 MB (10% inode=99%): [08:49:40] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:40] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:40] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:49:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [08:49:41] / on thyme is UNKNOWN: NRPE: Unable to read output [08:50:09] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:50:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:50:19] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:50:20] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:20] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:20] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:20] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:50:21] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:50:21] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:50] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [08:53:59] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [08:54:18] Load avg. on nightshade is OK: OK - load average: 11.57, 14.87, 13.44 [09:06:29] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14479.000000 [09:07:19] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14537.000000 [09:07:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14538.000000 [09:20:19] Load avg. on nightshade is WARNING: WARNING - load average: 23.82, 17.77, 15.04 [09:25:20] Load avg. on nightshade is OK: OK - load average: 9.11, 13.99, 14.31 [09:48:10] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [09:48:10] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 270029.000000 [09:48:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:10] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:48:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:48:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 336098.000000 [09:48:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 337508.000000 [09:48:30] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 266557.000000 [09:48:30] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:30] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:48:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:48:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:48:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:48:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:48:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 388611 MB (7% inode=63%): [09:48:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:49:01] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:49:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:49:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 375679.000000 [09:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [09:49:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descr [09:50:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180323.000000 [09:50:39] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62421 MB (10% inode=99%): [09:50:39] Load avg. on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:39] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:39] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:51:09] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:51:20] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:51:20] / on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:20] /tmp on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:20] NTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:20] Environment IPMI on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:51:21] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:21] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13175 [09:53:59] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16508 [09:54:29] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 8.966 second response time [10:01:50] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:01:50] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:02:30] SMTP on willow is OK: SMTP OK - 0.047 sec. response time [10:02:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:02:50] /tmp on willow is OK: DISK OK - free space: / 52805 MB (50% inode=99%): [10:02:50] / on willow is OK: DISK OK - free space: / 52805 MB (50% inode=99%): [10:02:50] Environment IPMI on willow is OK: ok: temperature ok fan ok voltage ok chassis ok [10:02:59] FMA on willow is OK: OK [10:03:09] Load avg. on willow is OK: OK - load average: 1.23, 1.60, 1.56 [10:03:09] NTP on willow is OK: NTP OK: Offset 0.004081 secs [10:07:19] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18137.000000 [10:07:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18138.000000 [10:07:30] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11245.000000 [10:48:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:48:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:48:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:48:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:48:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 388006 MB (7% inode=63%): [10:48:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:48:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:49:14] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 271236.000000 [10:49:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:14] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:49:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:49:14] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:49:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 340627.000000 [10:49:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 268841.000000 [10:49:29] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:49:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 377449.000000 [10:49:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 339111.000000 [10:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [10:49:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:50:39] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62124 MB (10% inode=99%): [10:50:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181409.000000 [10:51:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:52:19] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:52:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:50] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4483 [10:54:59] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4845 [10:59:30] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.759 second response time [11:01:50] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:01:50] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:03:00] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3531 [11:03:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:04:39] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:07:19] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15787.000000 [11:07:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8260.000000 [11:07:59] MySQL slave on z-dat-s2-b is OK: Uptime: 6546 Threads: 16 Questions: 6554847 Slow queries: 153 Opens: 81829 Flush tables: 1 Open tables: 256 Queries per second avg: 1001.351 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1576 [11:08:30] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5110.000000 [11:10:59] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2157 [11:15:39] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 18.288 second response time [11:15:59] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2135 [11:15:59] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2170 [11:19:39] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.040 second response time [11:25:30] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3592.000000 [11:25:50] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3524 [11:31:19] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3487.000000 [11:34:20] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1748.000000 [11:34:29] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1648.000000 [11:34:50] MySQL slave on z-dat-s5-b is OK: Uptime: 8029 Threads: 7 Questions: 10802959 Slow queries: 60 Opens: 844 Flush tables: 1 Open tables: 256 Queries per second avg: 1345.492 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1615 [11:36:59] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3622 [11:42:39] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:45:00] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3651 [11:48:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:48:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:48:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:48:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 387685 MB (7% inode=63%): [11:48:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:49:10] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 271052.000000 [11:49:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:10] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:49:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:49:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:49:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 342337.000000 [11:49:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 342682.000000 [11:49:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 271570.000000 [11:49:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:29] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:49:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 377500.000000 [11:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [11:49:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [11:50:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:50:39] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61902 MB (10% inode=99%): [11:51:20] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180549.000000 [11:52:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:52:19] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:52:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:00] MySQL slave on z-dat-s6-a is OK: Uptime: 63404 Threads: 8 Questions: 39873952 Slow queries: 5302 Opens: 94117 Flush tables: 1 Open tables: 2968 Queries per second avg: 628.887 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1758 [11:54:00] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3500 [11:54:22] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3201.000000 [11:56:59] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3455 [11:57:19] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1785.000000 [12:01:49] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:01:50] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:02:00] MySQL slave on z-dat-s7-a is OK: Uptime: 63944 Threads: 10 Questions: 52797571 Slow queries: 4078 Opens: 139788 Flush tables: 1 Open tables: 3104 Queries per second avg: 825.684 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1640 [12:03:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:06:00] MySQL slave on z-dat-s3-a is OK: Uptime: 72403 Threads: 30 Questions: 84031234 Slow queries: 16583 Opens: 857070 Flush tables: 1 Open tables: 16384 Queries per second avg: 1160.604 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1720 [12:13:10] hello all [12:18:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:19:30] SSH on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:35] Morning. [12:43:38] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:48:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:48:49] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:48:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:48:49] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 387368 MB (7% inode=63%): [12:48:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:49:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 340870.000000 [12:49:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 342690.000000 [12:49:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 271028.000000 [12:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [12:49:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:50:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 272478.000000 [12:50:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:50:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:50:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:50:29] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:50:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 378242.000000 [12:50:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61561 MB (10% inode=99%): [12:51:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:51:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180858.000000 [12:52:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:52:19] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [12:52:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:19] SSH on z-dat-s1-b is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [13:02:49] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [13:02:50] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [13:04:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:21:09] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=7900&oldid=7663&rcid=21746 * Edin * (+87) (/* {{#language:bs}} */ ) [13:22:26] [[User talk:Edin]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7901&rcid=21747 * Edin * (+71) (Created page with "You can contact me at [[:w:bs:Razgovor sa korisnikom:Edinwiki|bswiki]].") [13:33:22] @replag [13:33:22] Betacommand: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 1m 56s [-]; s1-user-wd: 4d 9h 27m 36s [-]; s2-rr: 14s [-]; s2-user: 14s [-]; s2-user-c: error; s2-user-wd: 3d 22h 59m 5s [-] [13:33:22] Betacommand: s3-user: 41s [-]; s3-user-wd: 3d 22h 27m 1s [-]; s4-user-wd: 3d 3h 54m 10s [-]; s5-rr-a-wd: 2d 2h 19m 32s [-]; s5-user: 14m 48s [-]; s5-user-c: 28m 30s [-]; s6-user-wd: 3d 3h 9m 7s [-]; s7-user-wd: 3d 22h 20m 51s [-] [13:34:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 30691 MB (3% inode=99%): [13:36:22] Looks like S1 is in trouble, I can't connect [13:41:18] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2121.000000 [13:43:40] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:47:33] DaBPunkt: can you take a S1 [13:48:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:48:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:48:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:48:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:48:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 387030 MB (7% inode=63%): [13:48:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:49:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 339247.000000 [13:49:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 341814.000000 [13:49:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 270178.000000 [13:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [13:49:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [13:50:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 273674.000000 [13:50:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:10] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:50:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:50:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:50:20] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1609.000000 [13:50:30] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:50:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 380215.000000 [13:50:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61307 MB (10% inode=99%): [13:51:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:51:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181296.000000 [13:52:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:52:19] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [13:52:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:50] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [14:02:50] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [14:04:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:18:55] who killed the mysql-process at z-dat-s1-b? [14:22:46] DaBPunkt: no clue, is that why S1 is down? [14:23:37] Betacommand: rosemary is up, so sql-s1 should work [14:23:53] what is not working? [14:25:02] enwiki-p.rrdb [14:25:53] [[Special:Log/newusers]] create 10 * Jmartin008 * (New user account) [14:26:15] m, strange [14:26:18] let me see [14:27:56] Betacommand: fixed [14:32:04] Marc A. Pelletier * [Toolserver-l] A (21) day in the Labs [14:43:40] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:46:04] DaBPunkt: thanks [14:48:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:48:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:48:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:48:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 386685 MB (7% inode=63%): [14:48:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [14:49:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descr [14:50:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 337828.000000 [14:50:30] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 270132.000000 [14:50:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 340549.000000 [14:50:30] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:50:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 382326.000000 [14:50:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:51:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 273800.000000 [14:51:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:51:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:51:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:51:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181257.000000 [14:51:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61014 MB (9% inode=99%): [14:52:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:52:19] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [14:52:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:01] @replag [14:53:01] Betacommand: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 5m 30s [+0.04 s/s]; s1-user-wd: 4d 10h 13m 39s [+0.58 s/s]; s2-rr: 15s [+0.00 s/s]; s2-user: 15s [+0.00 s/s]; s2-user-c: error; s2-user-wd: 3d 22h 35m 5s [-0.30 s/s] [14:53:02] Betacommand: s3-user: 41s [-]; s3-user-wd: 3d 21h 52m 44s [-0.43 s/s]; s4-user-wd: 3d 4h 3m 15s [+0.11 s/s]; s5-rr-a-wd: 2d 2h 20m 59s [+0.02 s/s]; s6-user-wd: 3d 3h 2m 8s [-0.09 s/s]; s7-user-wd: 3d 21h 49m 29s [-0.39 s/s] [14:53:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:50] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [15:02:50] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [15:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:18:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:18:47] @replag [15:18:48] Betacommand: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 13m 18s [+0.30 s/s]; s1-user-wd: 4d 10h 28m 6s [+0.56 s/s]; s2-user-c: error; s2-user-wd: 3d 22h 25m 21s [-0.38 s/s]; s3-user: 58s [+0.01 s/s]; s3-user-wd: 3d 21h 42m 15s [-0.41 s/s] [15:18:49] Betacommand: s4-user-wd: 3d 4h 1m 35s [-0.06 s/s]; s5-rr-a-wd: 2d 2h 20m 44s [-0.01 s/s]; s6-user-wd: 3d 3h 54s [-0.05 s/s]; s7-user-wd: 3d 21h 41m 19s [-0.32 s/s] [15:43:40] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:48:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [15:49:51] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:49:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:49:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [15:49:51] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 386424 MB (7% inode=63%): [15:49:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:50:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 274332.000000 [15:51:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:28] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 270254.000000 [15:51:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 336808.000000 [15:51:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 339630.000000 [15:51:30] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:51:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 384403.000000 [15:51:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:52:09] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:52:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:52:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:52:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:19] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [15:52:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:52:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181650.000000 [15:52:39] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60865 MB (9% inode=99%): [15:54:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:40] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 33820 MB (6% inode=99%): [16:03:50] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [16:03:51] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [16:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:15:04] Sumana Harihareswara * Re: [Toolserver-l] A (21) day in the Labs [16:18:19] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1887.000000 [16:18:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:18:50] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1874 [16:23:50] MySQL slave on rosemary is OK: Uptime: 1040366 Threads: 22 Questions: 931045886 Slow queries: 311445 Opens: 47534 Flush tables: 1 Open tables: 3390 Queries per second avg: 894.921 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1798 [16:24:19] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1795.000000 [16:29:04] Marc A. Pelletier * Re: [Toolserver-l] A (21) day in the Labs [16:31:22] (with that subject I hoped for a report of somebody who runs tools at the Labs…) [16:43:39] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:46:02] @replag [16:46:02] Merlissimo: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 19m 56s [+0.08 s/s]; s1-user-wd: 4d 11h 24m 1s [+0.64 s/s]; s2-rr: 12s [-0.00 s/s]; s2-user: 12s [-0.00 s/s]; s2-user-c: error; s2-user-wd: 3d 22h 30m 25s [+0.06 s/s] [16:46:03] Merlissimo: s3-user: 19s [-0.01 s/s]; s3-user-wd: 3d 21h 16m 12s [-0.30 s/s]; s4-user-wd: 3d 4h 38m 12s [+0.42 s/s]; s5-rr-a-wd: 2d 2h 40m 28s [+0.23 s/s]; s6-user-wd: 3d 3h 8m 46s [+0.09 s/s]; s7-user-wd: 3d 21h 16m 12s [-0.29 s/s] [16:48:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [16:49:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:49:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:50:31] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 386146 MB (7% inode=63%): [16:50:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:50:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:51:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 275971.000000 [16:51:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:30] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 340160.000000 [16:51:30] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 270649.000000 [16:51:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 335613.000000 [16:51:30] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:51:30] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 386824.000000 [16:52:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:52:10] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:52:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:52:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:52:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 182534.000000 [16:52:39] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60658 MB (9% inode=99%): [16:53:20] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:20] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [16:53:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:49] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [17:03:50] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [17:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:18:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:43:10] Betacommand: Thanks for pointing me in the python direction. That's some cool stuff [17:43:35] Now I just need my account approved [17:43:39] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:48:50] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1897 [17:48:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:49:20] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1916.000000 [17:49:50] / on thyme is UNKNOWN: NRPE: Unable to read output [17:49:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:49:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:50:29] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:50:50] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 385820 MB (7% inode=63%): [17:50:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:51:10] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 277419.000000 [17:52:10] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:10] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:52:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:52:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:52:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 335150.000000 [17:52:29] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 340609.000000 [17:52:29] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 270945.000000 [17:52:29] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:52:29] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 389214.000000 [17:52:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 183196.000000 [17:52:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60409 MB (9% inode=99%): [17:53:00] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:53:19] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:20] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [17:53:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:53:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:20] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:50] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [18:03:51] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [18:06:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:15:29] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420264 kB) free! [18:16:29] Free Memory on turnera is WARNING: WARNING - 5.4% (450056 kB) free! [18:17:30] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (401480 kB) free! [20:17:59] [[Special:Log/newusers]] create 10 * Spidy Hasan * (New user account)