[00:41:13] nacht ts [01:33:11] @replag [01:33:13] Betacommand: s1-rr-a: 10h 3m 58s [-]; s1-rr-a-wd: 2d 17h 47m 37s [-]; s1-user-wd: 4d 16h 46m 45s [-]; s2-user-c: error; s2-user-wd: 3d 22h 39m 19s [-]; s3-user: 12s [-]; s3-user-wd: 3d 21h 53s [-]; s4-user-wd: 3d 8h 38m 40s [-] [01:33:14] Betacommand: s5-rr-a: 10s [-]; s5-rr-a-wd: 2d 5h 13m 29s [-]; s5-user: 10s [-]; s6-user-wd: 3d 5h 34m 37s [-]; s7-user-wd: 3d 19h 54m 37s [-] [01:33:49] @replag [01:33:50] Betacommand: s1-rr-a: 10h 4m 12s [+0.37 s/s]; s1-rr-a-wd: 2d 17h 47m 39s [+0.05 s/s]; s1-user-wd: 4d 16h 46m 55s [+0.27 s/s]; s2-user-c: error; s2-user-wd: 3d 22h 39m 19s [-]; s3-user-wd: 3d 21h 37s [-0.44 s/s]; s4-user-wd: 3d 8h 39m 11s [+0.85 s/s]; s5-rr-a-wd: 2d 5h 13m 31s [+0.05 s/s] [01:33:51] Betacommand: s6-user-wd: 3d 5h 34m 57s [+0.55 s/s]; s7-user-wd: 3d 19h 54m 36s [-0.03 s/s] [03:35:24] Toolserver having trouble? [03:43:29] What happened to toolserver? [03:51:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:51:20] / on thyme is UNKNOWN: NRPE: Unable to read output [03:51:25] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [03:52:06] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:06] MySQL slave on daphne is OK: Uptime: 418108 Threads: 30 Questions: 164199460 Slow queries: 19629 Opens: 7529 Flush tables: 1 Open tables: 1361 Queries per second avg: 392.720 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1199 [03:52:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:52:36] Free Memory on damiana is OK: OK - 91.9% (7697408 kB) free. [03:52:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 295205.000000 [03:53:05] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 281486.000000 [03:53:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 193348.000000 [03:53:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57852 MB (9% inode=99%): [03:53:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 341186.000000 [03:54:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 331751.000000 [03:54:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 410100.000000 [03:54:57] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:27] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:55:57] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:56:37] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [03:57:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:57:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:57:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:57:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 383762 MB (7% inode=63%): [03:57:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:57:29] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [03:57:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:57:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:57:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:57:47] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:01:17] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 33512 MB (6% inode=99%): [04:04:18] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:04:57] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2212.000000 [04:04:57] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [04:05:06] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2219.000000 [04:05:36] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [04:06:27] SSH on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:12:37] SSH on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:12:57] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [04:18:17] SSH on nightshade is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [04:21:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:26:27] SSH on yarrow is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [04:26:47] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [04:28:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3652.000000 [04:28:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3656.000000 [04:29:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3659.000000 [04:50:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:51:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:51:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:51:16] / on thyme is UNKNOWN: NRPE: Unable to read output [04:51:27] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:52:48] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 298806.000000 [04:53:07] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 285087.000000 [04:53:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 196951.000000 [04:53:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57266 MB (9% inode=99%): [04:53:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:54:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 344787.000000 [04:54:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 335351.000000 [04:54:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 413701.000000 [04:54:57] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:39] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:55:57] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [04:57:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:57:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 383571 MB (7% inode=63%): [04:57:37] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [04:57:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:57:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:57:58] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:58:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:58:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:58:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:05:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:05:38] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [05:21:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:28:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7254.000000 [05:28:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7256.000000 [05:30:06] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7320.000000 [05:50:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:51:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:51:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:51:17] / on thyme is UNKNOWN: NRPE: Unable to read output [05:51:26] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:52:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 302410.000000 [05:53:07] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 288688.000000 [05:53:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 200551.000000 [05:53:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57000 MB (9% inode=99%): [05:54:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 338952.000000 [05:54:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348388.000000 [05:54:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:54:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 417301.000000 [05:54:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:36] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:57:39] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 383372 MB (7% inode=63%): [05:57:39] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:57:39] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:57:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:57:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:57:40] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [05:58:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:58:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:58:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:58:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:59:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:05:16] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:05:38] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:21:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:21:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:29:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10914.000000 [06:29:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10915.000000 [06:30:06] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10920.000000 [06:51:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:52:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:52:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:52:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:52:17] / on thyme is UNKNOWN: NRPE: Unable to read output [06:52:25] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 306010.000000 [06:53:07] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 292288.000000 [06:53:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 204150.000000 [06:53:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56798 MB (9% inode=99%): [06:54:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 351988.000000 [06:54:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 342552.000000 [06:54:06] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:54:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 420902.000000 [06:54:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:57:06] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 383136 MB (7% inode=63%): [06:58:37] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:58:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:58:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:58:37] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:58:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:59:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:59:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:59:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:59:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:59:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:05:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:06:37] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [07:21:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:22:36] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:29:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14514.000000 [07:29:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14516.000000 [07:30:06] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14520.000000 [07:51:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:52:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:52:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:52:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:52:17] / on thyme is UNKNOWN: NRPE: Unable to read output [07:52:27] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 309376.000000 [07:53:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207233.000000 [07:53:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 295893.000000 [07:53:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56630 MB (9% inode=99%): [07:53:37] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [07:54:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355012.000000 [07:54:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 346149.000000 [07:54:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:54:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 424147.000000 [07:54:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:38] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:57:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:37] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:58:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 382969 MB (7% inode=63%): [07:58:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:58:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:58:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:58:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:59:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:59:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:00:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:05:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:06:46] aliasd on yarrow is CRITICAL: Connection refused [08:06:56] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [08:11:38] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2117 [08:12:56] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3559.000000 [08:13:37] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2163 [08:14:55] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1595.000000 [08:17:36] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2085 [08:19:37] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [08:21:56] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3290.000000 [08:22:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:24:56] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1681.000000 [08:26:37] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2639 [08:30:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18121.000000 [08:35:38] MySQL slave on z-dat-s3-a is OK: Uptime: 45353 Threads: 25 Questions: 70110415 Slow queries: 4092 Opens: 725101 Flush tables: 1 Open tables: 16381 Queries per second avg: 1545.882 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1674 [08:45:38] MySQL slave on z-dat-s7-a is OK: Uptime: 44266 Threads: 5 Questions: 28433903 Slow queries: 2060 Opens: 54533 Flush tables: 1 Open tables: 1255 Queries per second avg: 642.341 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [08:51:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:51:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:52:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:52:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:52:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:52:17] / on thyme is UNKNOWN: NRPE: Unable to read output [08:52:36] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 299406.000000 [08:53:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 206846.000000 [08:53:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56423 MB (9% inode=99%): [08:53:37] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [08:53:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 310638.000000 [08:54:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354346.000000 [08:54:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348176.000000 [08:54:06] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 425279.000000 [08:54:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:38] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:57:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:59:36] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [08:59:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 382526 MB (7% inode=63%): [08:59:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:59:37] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [08:59:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:59:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:59:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:00:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:05:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:06:46] aliasd on yarrow is CRITICAL: Connection refused [09:06:57] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [09:23:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:30:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21721.000000 [09:51:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:52:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:52:36] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:53:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:53:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:53:17] / on thyme is UNKNOWN: NRPE: Unable to read output [09:53:38] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [09:53:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 313084.000000 [09:54:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347168.000000 [09:54:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 353559.000000 [09:54:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 427149.000000 [09:54:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 303065.000000 [09:54:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207845.000000 [09:54:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56192 MB (9% inode=99%): [09:54:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:57:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:59:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:59:37] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [09:59:37] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [09:59:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 382101 MB (7% inode=63%): [09:59:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:59:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:59:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:59:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:00:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:05:16] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:06:46] aliasd on yarrow is CRITICAL: Connection refused [10:07:56] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:23:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:30:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25321.000000 [10:51:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:52:36] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:53:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:53:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:53:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:53:17] / on thyme is UNKNOWN: NRPE: Unable to read output [10:54:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 306666.000000 [10:54:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207380.000000 [10:54:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55974 MB (9% inode=99%): [10:54:37] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:54:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 314684.000000 [10:55:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 346309.000000 [10:55:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 352645.000000 [10:55:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 429501.000000 [10:55:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:58:06] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:59:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:00:13] @replag [11:00:13] Betacommand: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 52s [-]; s1-user-wd: 4d 23h 20m 34s [+0.58 s/s]; s2-rr: 7h 32m 15s [+1.00 s/s]; s2-user: 7h 32m 15s [+1.00 s/s]; s2-user-c: error; s2-user-wd: 4d 1h 56m 55s [-0.23 s/s] [11:00:14] Betacommand: s3-user: 27s [-]; s3-user-wd: 4d 1h 1m 51s [-0.26 s/s]; s4-user-wd: 3d 15h 26m 35s [+0.55 s/s]; s5-rr-a-wd: 2d 9h 34m 52s [+0.06 s/s]; s5-user: 7h 32m 28s [+1.00 s/s]; s6-user-wd: 3d 13h 17m 12s [+1.00 s/s]; s7-user-wd: 4d 11m 20s [-0.25 s/s] [11:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:00:38] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [11:00:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [11:00:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 381835 MB (7% inode=63%): [11:00:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:00:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:00:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:01:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:05:18] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:06:46] aliasd on yarrow is CRITICAL: Connection refused [11:07:57] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:24:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:30:06] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28921.000000 [11:51:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:52:37] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:54:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:54:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 310266.000000 [11:54:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 206386.000000 [11:54:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55715 MB (9% inode=99%): [11:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:54:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:54:17] / on thyme is UNKNOWN: NRPE: Unable to read output [11:54:38] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:55:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 346257.000000 [11:55:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 352102.000000 [11:55:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 432143.000000 [11:55:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 315422.000000 [11:55:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:57:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:58:06] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:00:38] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [12:00:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [12:00:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 381015 MB (7% inode=63%): [12:00:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:00:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:00:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:00:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:01:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:05:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:07:46] aliasd on yarrow is CRITICAL: Connection refused [12:08:56] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:24:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:29:57] Load avg. on daphne is WARNING: WARNING - load average: 16.79, 15.85, 14.38 [12:30:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32521.000000 [12:32:56] Load avg. on daphne is OK: OK - load average: 13.57, 14.93, 14.30 [12:52:36] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:54:06] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 313866.000000 [12:54:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 205022.000000 [12:54:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55530 MB (9% inode=99%): [12:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:54:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [12:54:17] / on thyme is UNKNOWN: NRPE: Unable to read output [12:55:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 346488.000000 [12:55:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 352547.000000 [12:55:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:55:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 435273.000000 [12:55:37] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:55:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 316105.000000 [12:55:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:40] @replag [12:56:40] Susan: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 9m 48s [+0.08 s/s]; s1-user-c: 17m 43s [-]; s1-user-wd: 5d 55m 56s [+0.82 s/s]; s2-rr: 9h 28m 42s [+1.00 s/s]; s2-user: 9h 28m 42s [+1.00 s/s]; s2-user-c: error [12:56:41] Susan: s2-user-wd: 4d 1h 55m 31s [-0.01 s/s]; s3-user-wd: 4d 55m 32s [-0.05 s/s]; s4-user-wd: 3d 15h 48m 35s [+0.19 s/s]; s5-rr-a-wd: 2d 8h 55m 21s [-0.34 s/s]; s5-user: 9h 28m 55s [+1.00 s/s]; s6-user-wd: 3d 15h 13m 39s [+1.00 s/s]; s7-user-wd: 4d 14m 59s [+0.03 s/s] [12:57:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:58:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:01:38] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [13:01:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [13:01:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 380796 MB (7% inode=63%): [13:01:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:01:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:01:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:01:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:01:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:05:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:07:46] aliasd on yarrow is CRITICAL: Connection refused [13:08:56] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [13:12:37] MySQL on z-dat-s1-b is OK: Uptime: 19490 Threads: 5 Questions: 611 Slow queries: 0 Opens: 49 Flush tables: 1 Open tables: 42 Queries per second avg: 0.31 [13:18:57] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2104.000000 [13:21:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:24:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:30:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36121.000000 [13:36:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 29306 MB (2% inode=99%): [13:39:57] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1769.000000 [13:52:37] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:54:06] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 317466.000000 [13:54:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 202636.000000 [13:54:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55249 MB (9% inode=99%): [13:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:54:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:54:17] / on thyme is UNKNOWN: NRPE: Unable to read output [13:55:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 346898.000000 [13:55:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 353549.000000 [13:55:06] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 438076.000000 [13:55:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:55:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 316593.000000 [13:55:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:59:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:59:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:00:42] Hello all [14:01:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:01:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:02:36] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [14:02:36] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [14:02:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 380383 MB (7% inode=63%): [14:02:36] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:02:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:02:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:02:56] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1970.000000 [14:05:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:08:46] aliasd on yarrow is CRITICAL: Connection refused [14:08:57] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23143 [14:16:56] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1786.000000 [14:21:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:24:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:30:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39721.000000 [14:33:57] Load avg. on daphne is WARNING: WARNING - load average: 17.63, 16.46, 14.59 [14:52:37] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:06] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 321065.000000 [14:54:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 201753.000000 [14:54:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54997 MB (9% inode=99%): [14:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:54:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:54:17] / on thyme is UNKNOWN: NRPE: Unable to read output [14:54:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:55:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 353746.000000 [14:55:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347397.000000 [14:55:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:55:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 440574.000000 [14:56:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 316886.000000 [14:56:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:00:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:00:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:01:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:01:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:02:38] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [15:02:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [15:02:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:02:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:02:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 380382 MB (7% inode=63%): [15:02:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:05:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:08:47] aliasd on yarrow is CRITICAL: Connection refused [15:08:57] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10341 [15:21:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:25:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:26:56] Load avg. on daphne is OK: OK - load average: 13.42, 14.42, 14.95 [15:31:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43381.000000 [15:37:08] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 103960 MB (10% inode=99%): [15:37:57] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1969.000000 [15:38:17] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1975 [15:50:07] /sql on z-dat-s1-b is OK: DISK OK - free space: /sql 112300 MB (11% inode=99%): [15:51:57] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3547 [15:52:37] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:55:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 324725.000000 [15:55:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 200176.000000 [15:55:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54735 MB (8% inode=99%): [15:55:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:55:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:55:17] / on thyme is UNKNOWN: NRPE: Unable to read output [15:56:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347617.000000 [15:56:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354412.000000 [15:56:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:56:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 442200.000000 [15:56:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 317224.000000 [15:56:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:00:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:00:06] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:01:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:02:16] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 33209 MB (6% inode=99%): [16:02:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:03:36] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [16:03:37] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [16:03:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 380150 MB (7% inode=63%): [16:03:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:03:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:03:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:06:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:09:51] aliasd on yarrow is CRITICAL: Connection refused [16:15:57] MySQL slave on z-dat-s1-b is OK: Uptime: 30491 Threads: 8 Questions: 10374421 Slow queries: 259 Opens: 293 Flush tables: 1 Open tables: 220 Queries per second avg: 340.245 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1758 [16:21:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:26:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:31:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46981.000000 [16:38:17] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3448 [16:38:57] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3483.000000 [16:40:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3603.000000 [16:41:16] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628 [16:52:37] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:55:06] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355115.000000 [16:56:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347759.000000 [16:56:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 441154.000000 [16:56:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:56:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 326152.000000 [16:56:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 199081.000000 [16:56:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54484 MB (8% inode=99%): [16:56:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:56:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:56:18] / on thyme is UNKNOWN: NRPE: Unable to read output [16:56:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 318052.000000 [16:59:38] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:00:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:00:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:01:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:01:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:02:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:03:37] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [17:03:37] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [17:03:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:03:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 379712 MB (7% inode=63%): [17:03:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:03:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:06:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:09:46] aliasd on yarrow is CRITICAL: Connection refused [17:21:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:27:36] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:31:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50581.000000 [17:40:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6403.000000 [17:41:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6410 [17:43:23] is there any description regarding user_properties_anonym table? [17:53:37] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:56:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347413.000000 [17:57:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355131.000000 [17:57:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 440069.000000 [17:57:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:57:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 325847.000000 [17:57:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 198025.000000 [17:57:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54198 MB (8% inode=99%): [17:57:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:57:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:57:18] / on thyme is UNKNOWN: NRPE: Unable to read output [17:57:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 318255.000000 [18:00:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:00:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:01:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:01:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:01:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:02:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:03:37] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [18:03:37] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [18:04:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 378760 MB (7% inode=63%): [18:04:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:04:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:04:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:04:53] Hi! I'm trying to do a replace like "\{\{Гфдл\}\}" "{{GFDL-sr}}" in cronie but it only works if I copy the code into the command prompt on toolserver. I thing it is a coding problem so perhaps someone can tell me an easy way to fix that? [18:05:42] think* And you can see the full code in http://toolserver.org/~mga73/queries/cron/test.sh [18:07:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:09:46] aliasd on yarrow is CRITICAL: Connection refused [18:16:45] Susan: around? [18:21:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:23:05] DaB [18:23:16] gimme some access! [18:27:36] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:31:06] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54181.000000 [18:32:33] MGA73: { is no special char in bash, so why to you escape it? [18:34:08] DaBPunkt: Hm... Guess I'm used to escape [ ( . etc so therefore I also escape { [18:35:11] these are also no special chars in bash (but in regex) [18:35:33] so where excactly is the problem? [18:37:15] I'm trying to change variants of GFDL-templates to the standard GFDL.. But I can't change {{Гфдл}} to {{GFDL-sr}} because Гфдл is not standard caracters [18:38:09] I can do it if I run it directly but with SGE it does not work... (see http://toolserver.org/~mga73/queries/cron/test.sh ) [18:38:46] MGA73: I saw the file but where is the output or the error-message? [18:39:27] http://toolserver.org/~mga73/queries/cron/test.log but it just says "No changes were necessary in [[User:MGA73/Sandbox]]" [18:39:38] Does it work if you commit the file to SGE on the shell? [18:40:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7112.000000 [18:41:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7088 [18:42:51] DaBPunkt: At the moment I get "Using keyboard-interactive authentication." when I want to login [18:43:24] MGA73: which host? [18:43:31] willow [18:43:39] Ah cool... I got in [18:47:07] Okay running the script with "qcronsub -N test-script test.sh" gives me these http://commons.wikimedia.org/w/index.php?title=User%3AMGA73%2FSandbox&diff=94273028&oldid=94270518 so it changes the latin letters but not the "funny" ones [18:47:34] that's cyril afais [18:50:02] MGA73: where is the script in your home? [18:51:28] You mean like: mga73@willow:~/queries/cron$ ? [18:53:38] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:48] MGA73: add a "LC_ALL=en_US.UTF-8" to the file for testing [18:55:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:56:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355235.000000 [18:57:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347362.000000 [18:57:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 438980.000000 [18:57:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:57:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:18] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 324602.000000 [18:57:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 197186.000000 [18:57:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53917 MB (8% inode=99%): [18:57:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:57:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:57:19] / on thyme is UNKNOWN: NRPE: Unable to read output [18:57:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 318158.000000 [19:00:07] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:00:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:00:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:01:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:01:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:02:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:02:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:03:33] DaBPunkt: Okay just adding the code into the http://toolserver.org/~mga73/queries/cron/test.sh does not work. Perhaps you could help me some more? [19:03:36] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [19:03:36] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [19:04:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 374355 MB (7% inode=63%): [19:04:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:04:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:04:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:07:17] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:08:07] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.009 second response time [19:08:57] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [19:09:47] aliasd on yarrow is CRITICAL: Connection refused [19:09:52] MGA73: sorry, at the moment I' busy with non-TS-stuff. Tomorrow would be a good time [19:10:43] Okay... :-) [19:21:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:27:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:31:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57781.000000 [19:41:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6942 [19:41:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6951.000000 [19:44:46] aliasd on yarrow is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [19:48:07] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 10.672 second response time [19:54:36] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:56:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347930.000000 [19:57:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354999.000000 [19:57:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 439068.000000 [19:57:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:57:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 318570.000000 [19:57:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 321631.000000 [19:58:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 197318.000000 [19:58:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53630 MB (8% inode=99%): [19:58:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:58:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:58:17] / on thyme is UNKNOWN: NRPE: Unable to read output [20:00:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:00:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:01:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:01:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:02:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:02:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:04:26] aliasd on mayapple is OK: TCP OK - 0.020 second response time on port 984 [500 Not found.] [20:04:37] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [20:04:37] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [20:04:37] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:04:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 377121 MB (7% inode=63%): [20:04:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:04:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:28:36] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:32:06] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61441.000000 [20:41:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6354.000000 [20:42:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6366 [20:51:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:56:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:57:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 320031.000000 [20:57:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 354991.000000 [20:58:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348379.000000 [20:58:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 438460.000000 [20:58:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:58:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 319719.000000 [20:58:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 197155.000000 [20:58:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53436 MB (8% inode=99%): [20:58:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:58:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:58:18] / on thyme is UNKNOWN: NRPE: Unable to read output [21:00:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:00:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:02:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:02:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:02:47] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:02:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:04:36] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [21:04:36] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [21:04:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 376834 MB (7% inode=63%): [21:04:36] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:04:36] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:04:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:28:36] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:32:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65041.000000 [21:41:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5634.000000 [21:43:16] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5655 [21:51:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:56:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:57:04] Горан Обрадовић * [Toolserver-l] srwiki_p database incosistency [21:57:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 322837.000000 [21:57:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348667.000000 [21:58:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355549.000000 [21:58:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 437830.000000 [21:58:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:58:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:58:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 317356.000000 [21:58:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 197224.000000 [21:58:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53116 MB (8% inode=99%): [21:58:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:58:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:58:18] / on thyme is UNKNOWN: NRPE: Unable to read output [22:00:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:01:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:02:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:03:07] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:03:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:03:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:04:38] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [22:04:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [22:04:38] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 376587 MB (7% inode=63%): [22:04:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:04:38] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:04:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:08:07] Free Memory on turnera is WARNING: WARNING - 5.1% (429992 kB) free! [22:09:07] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (393668 kB) free! [22:19:08] Free Memory on turnera is OK: OK - 10.7% (895808 kB) free. [22:28:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:32:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68641.000000 [22:41:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5593.000000 [22:43:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5585 [22:51:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:56:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:57:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 325068.000000 [22:57:58] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:06] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (367540 kB) free! [22:58:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 316143.000000 [22:59:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 356290.000000 [22:59:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348476.000000 [22:59:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 437508.000000 [22:59:07] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:59:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:17] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 197844.000000 [22:59:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52825 MB (8% inode=99%): [22:59:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:59:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:59:17] / on thyme is UNKNOWN: NRPE: Unable to read output [23:02:00] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:02:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:03:09] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:30] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:03:50] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:04:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:04:10] Free Memory on turnera is OK: OK - 86.9% (7279872 kB) free. [23:04:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:04:49] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [23:05:10] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [23:05:30] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 374481 MB (7% inode=63%): [23:05:30] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:05:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:28:50] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:32:10] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72241.000000 [23:37:00] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2211.000000 [23:37:00] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2211.000000 [23:38:54] Alchimista: Hi. [23:39:55] hey susan, nice to hear about you. Are you familiar with the user_properties_anonym table? I have some questions about it, and notice that you use it on some scripts [23:41:18] what is the problem with that table? [23:41:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [23:42:50] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3594 [23:43:00] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3590.000000 [23:54:21] DaBPunkt: i'm tryng to find a description of the fields, for some stats [23:55:55] ask in #mediawiki [23:57:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:57:13] at the TS the up_user-field is missing for that table – for exchange we have the ts_user_touched_cropped-field [23:57:49] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 325802.000000 [23:58:00] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:58:20] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 319621.000000 [23:58:30] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:58:56] Alchimista: You can also ask in here. :-) [23:59:02] Alchimista: But you probably just want... [23:59:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 359837.000000 [23:59:10] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 440892.000000 [23:59:10] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:16] Alchimista: https://www.mediawiki.org/wiki/Manual:User_properties_table [23:59:19] / on thyme is UNKNOWN: NRPE: Unable to read output [23:59:29] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 201420.000000 [23:59:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:59:34] Alchimista: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=blob;f=maintenance/tables.sql;hb=HEAD is a good link to bookmark. ;-) [23:59:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:59:50] RAID on thyme is UNKNOWN: NRPE: Unable to read output