[00:27:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:27:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:27:29] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 362861 MB (6% inode=63%): [00:27:29] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:27:29] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:27:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:27:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:27:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:27:38] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [00:27:48] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 384406.000000 [00:27:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:27:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:27:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:27:58] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33088.000000 [00:28:08] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52627.000000 [00:28:08] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 393437.000000 [00:28:08] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 372371.000000 [00:28:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 476312.000000 [00:28:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:28:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 252699.000000 [00:28:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47293 MB (7% inode=99%): [00:28:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:28:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:28:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:28:18] / on thyme is UNKNOWN: NRPE: Unable to read output [00:28:28] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:28:28] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:33:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:48] aliasd on yarrow is CRITICAL: Connection refused [00:33:57] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:58] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 403847.000000 [00:34:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:38:42] jeremyb_: Yeah, I noticed the moment I pressed Send :-). [00:45:04] Marc A. Pelletier * Re: [Toolserver-l] When is the best time of day to run programs? [00:58:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:04:03] @replag [01:04:04] Alchimista: s1-rr-a: 15h 13m 13s [+1.00 s/s]; s1-rr-a-wd: 3d 22h 15m 54s [-0.05 s/s]; s1-user: 9h 47m 43s [+1.00 s/s]; s1-user-c: 12s [-]; s1-user-wd: 5d 12h 22m 22s [-0.09 s/s]; s2-rr: 15h 13m 17s [+1.00 s/s]; s2-user: 15h 13m 17s [+1.00 s/s]; s2-user-c: error [01:04:05] Alchimista: s2-user-wd: 4d 16h 9m 0s [-0.29 s/s]; s3-user: 2m 20s [+0.01 s/s]; s3-user-wd: 4d 7h 8m 25s [+0.16 s/s]; s4-rr-a: 12s [-]; s4-user: 12s [-]; s4-user-wd: 4d 11h 10m 42s [+0.85 s/s]; s5-rr-a-wd: 2d 22h 16m 26s [+0.20 s/s]; s5-user: 15h 13m 15s [+1.00 s/s] [01:04:06] Alchimista: s5-user-c: 12s [-0.73 s/s]; s6-user-wd: 4d 7h 30m 32s [+0.15 s/s]; s7-user-wd: 4d 13h 21m 9s [+0.02 s/s] [01:11:22] Three messages? [01:11:24] My goodness. [01:27:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:27:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:27:29] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 362706 MB (6% inode=63%): [01:27:29] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:27:29] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:27:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:27:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:27:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [01:27:38] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [01:27:48] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 386666.000000 [01:27:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:27:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:27:58] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:27:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36688.000000 [01:28:08] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56227.000000 [01:28:08] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 393665.000000 [01:28:08] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 372809.000000 [01:28:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 476746.000000 [01:28:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 253235.000000 [01:28:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47032 MB (7% inode=99%): [01:28:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:28:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:28:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:28:19] / on thyme is UNKNOWN: NRPE: Unable to read output [01:28:28] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:28:28] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:29:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:29:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:33:47] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:47] aliasd on yarrow is CRITICAL: Connection refused [01:33:58] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:33:58] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 404484.000000 [01:34:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:58:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:27:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:27:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:27:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 360919 MB (6% inode=63%): [02:27:38] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:27:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:27:38] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [02:27:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [02:27:48] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 389084.000000 [02:27:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:27:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:27:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:27:58] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40289.000000 [02:28:07] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59827.000000 [02:28:07] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 393814.000000 [02:28:08] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 373358.000000 [02:28:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 477237.000000 [02:28:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 253756.000000 [02:28:19] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46860 MB (7% inode=99%): [02:28:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:28:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [02:28:19] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:28:19] / on thyme is UNKNOWN: NRPE: Unable to read output [02:28:28] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:28:28] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:28:28] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [02:29:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:29:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:31:37] Free Memory on damiana is WARNING: WARNING - 5.6% (467720 kB) free! [02:33:17] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 31597 MB (5% inode=99%): [02:33:38] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421548 kB) free! [02:33:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:48] aliasd on yarrow is CRITICAL: Connection refused [02:33:58] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:58] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:34:08] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 404410.000000 [02:34:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:34:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:34:38] Free Memory on damiana is WARNING: WARNING - 5.6% (471088 kB) free! [02:46:38] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (409824 kB) free! [02:58:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:01:38] [[Special:Log/newusers]] create 10 * Thenopen * (New user account) [03:10:37] Free Memory on damiana is WARNING: WARNING - 6.0% (500040 kB) free! [03:21:38] Free Memory on damiana is CRITICAL: CRITICAL - 3.9% (325056 kB) free! [03:27:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:27:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:27:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 361356 MB (6% inode=63%): [03:27:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:27:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:27:38] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [03:27:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [03:27:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 390491.000000 [03:27:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:27:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:27:57] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:27:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43888.000000 [03:28:08] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63427.000000 [03:28:08] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 394979.000000 [03:28:08] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 374980.000000 [03:28:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 477869.000000 [03:28:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 254397.000000 [03:28:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46720 MB (7% inode=99%): [03:28:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:28:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:28:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:28:28] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:28:28] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:28:28] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [03:29:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:29:17] / on thyme is UNKNOWN: NRPE: Unable to read output [03:29:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:33:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:48] aliasd on yarrow is CRITICAL: Connection refused [03:33:58] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:58] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:34:08] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 405515.000000 [03:34:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:34:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:44:18] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 32587 MB (6% inode=99%): [03:46:17] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 31361 MB (5% inode=99%): [04:21:38] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (126608 kB) free! [04:27:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:27:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:27:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 362185 MB (6% inode=63%): [04:27:37] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:27:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:27:38] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [04:27:38] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [04:27:49] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 392999.000000 [04:27:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:27:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:27:58] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:27:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:27:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47488.000000 [04:28:08] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67029.000000 [04:28:08] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 396123.000000 [04:28:08] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 375798.000000 [04:28:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 478294.000000 [04:28:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:28:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 254520.000000 [04:28:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46584 MB (7% inode=99%): [04:28:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:28:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:28:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [04:28:28] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:28:28] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:28:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:28:28] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [04:29:08] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:29:17] / on thyme is UNKNOWN: NRPE: Unable to read output [04:29:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:33:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:48] aliasd on yarrow is CRITICAL: Connection refused [04:33:57] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:57] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:07] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:07] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 406050.000000 [04:34:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:17:45] Free Memory on damiana is OK: OK - 83.4% (6989576 kB) free. [05:20:03] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:04] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:04] Federico Leva (Nemo) * Re: [Toolserver-l] Can't log into toolserver [05:27:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:27:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:27:46] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 362114 MB (6% inode=63%): [05:27:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:27:55] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:27:55] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [05:27:55] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [05:27:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:28:37] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 396074.000000 [05:28:38] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:28:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51126.000000 [05:28:39] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 398211.000000 [05:28:39] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 479993.000000 [05:28:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:28:39] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70659.000000 [05:28:39] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 377518.000000 [05:28:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46452 MB (7% inode=99%): [05:28:47] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:28:48] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 256363.000000 [05:28:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:28:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:28:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:28:48] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [05:28:48] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:29:02] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:29:03] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:29:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:29:40] / on thyme is UNKNOWN: NRPE: Unable to read output [05:30:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:34:04] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:34:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:34:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:34:40] aliasd on yarrow is CRITICAL: Connection refused [05:34:41] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 408109.000000 [05:34:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:35:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:38] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [05:40:38] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [05:50:18] @replag [05:50:19] liangent: s1-rr-a: 19h 59m 27s [-]; s1-rr-a-wd: 3d 21h 38m 30s [-]; s1-user: 14h 33m 57s [-]; s1-user-wd: 5d 13h 41m 41s [-]; s2-rr: 19h 59m 31s [-]; s2-user: 19h 59m 31s [-]; s2-user-c: error; s2-user-wd: 4d 17h 37m 37s [-] [05:50:20] liangent: s3-user: 1m 44s [-]; s3-user-wd: 4d 9h 30s [-]; s4-user-wd: 4d 14h 23m 7s [-]; s5-rr-a-wd: 2d 23h 34m 22s [-]; s5-user: 19h 59m 29s [-]; s5-user-c: 46m 45s [-]; s6-user-wd: 4d 9h 13m 46s [-]; s7-user: 11s [-] [05:50:21] liangent: s7-user-wd: 4d 14h 58m 39s [-] [05:58:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:04:36] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [06:04:37] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [06:20:05] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:08] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:27:56] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:27:56] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:27:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:28:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:28:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:28:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:28:46] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 361822 MB (6% inode=63%): [06:28:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:28:56] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:29:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:05] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:39] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 399734.000000 [06:29:39] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:29:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54786.000000 [06:29:39] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 483653.000000 [06:29:39] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 401871.000000 [06:29:39] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74319.000000 [06:29:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:29:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46259 MB (7% inode=99%): [06:29:41] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 381179.000000 [06:29:41] / on thyme is UNKNOWN: NRPE: Unable to read output [06:29:46] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:29:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 260024.000000 [06:29:46] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:29:46] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:29:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:30:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:30:39] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:31:46] SSH on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:06] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:06] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:35:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:35:38] aliasd on yarrow is CRITICAL: Connection refused [06:35:39] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 411769.000000 [06:58:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:04:38] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7255.000000 [07:04:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7254.000000 [07:11:39] aliasd on yarrow is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [07:16:38] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2696.000000 [07:17:39] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1764.000000 [07:18:56] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2145 [07:20:04] aliasd on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:04] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:35] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3392.000000 [07:27:56] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:27:56] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 76026 [07:27:56] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2177 [07:27:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:28:05] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75049 [07:28:05] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:28:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:28:56] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:28:56] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2135 [07:29:05] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:30:15] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:30:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 262476.000000 [07:30:15] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:30:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:30:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 361567 MB (6% inode=63%): [07:30:16] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 76353 [07:30:16] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:30:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:30:37] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:30:37] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 402762.000000 [07:30:37] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1631.000000 [07:30:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58446.000000 [07:30:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 486130.000000 [07:30:38] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76271.000000 [07:30:38] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 405522.000000 [07:30:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:30:39] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46064 MB (7% inode=99%): [07:30:40] / on thyme is UNKNOWN: NRPE: Unable to read output [07:30:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 384811.000000 [07:31:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:31:38] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:31:46] SSH on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:35:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:05] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:39] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 414622.000000 [07:44:03] Dr. Trigon * Re: [Toolserver-l] Login and account [07:44:57] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [07:46:03] Dr. Trigon * Re: [Toolserver-l] mayapple config & cron jobs [07:54:03] Marlen Caemmerer * Re: [Toolserver-l] Login and account [07:55:58] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3635 [07:58:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:17:58] MySQL slave on z-dat-s6-a is OK: Uptime: 172687 Threads: 9 Questions: 121863968 Slow queries: 10784 Opens: 251757 Flush tables: 1 Open tables: 3108 Queries per second avg: 705.692 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1624 [08:20:57] aliasd on mayapple is CRITICAL: No data received from host [08:21:04] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:57] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3351 [08:26:57] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3572 [08:27:36] SSH on sage is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [08:28:09] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 71134 [08:28:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:28:56] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:28:56] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 74272 [08:28:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:29:46] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:29:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 262809.000000 [08:29:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:29:46] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 361231 MB (6% inode=63%): [08:29:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:29:47] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 74042 [08:29:51] @replag [08:29:51] DarkoNeko: s1-rr-a: 20h 37m 25s [+0.24 s/s]; s1-rr-a-wd: 3d 22h 48m 17s [+0.44 s/s]; s1-user: 17h 13m 30s [+1.00 s/s]; s1-user-c: 2m 3s [-]; s1-user-wd: 5d 15h 18m 46s [+0.61 s/s]; s2-rr: 19h 47m 22s [-0.08 s/s]; s2-user: 19h 47m 22s [-0.08 s/s]; s2-user-c: error [08:29:52] DarkoNeko: s2-user-wd: 4d 19h 41m 25s [+0.78 s/s]; s3-user: 48m 44s [+0.29 s/s]; s3-user-wd: 4d 11h 29m 0s [+0.93 s/s]; s4-user-wd: 4d 16h 40m 23s [+0.86 s/s]; s5-rr-a-wd: 3d 1h 11s [+0.54 s/s]; s5-user: 20h 34m 8s [+0.22 s/s]; s6-user-wd: 4d 11h 30m 24s [+0.86 s/s]; s7-user: 42m 10s [+0.26 s/s] [08:29:53] DarkoNeko: s7-user-wd: 4d 17h 23m 15s [+0.91 s/s] [08:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:29:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:29:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:29:58] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:30:04] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:38] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:30:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 405647.000000 [08:30:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62046.000000 [08:30:38] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74064.000000 [08:30:38] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 408194.000000 [08:30:39] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 487129.000000 [08:30:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:30:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45837 MB (7% inode=99%): [08:30:40] / on thyme is UNKNOWN: NRPE: Unable to read output [08:31:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:31:38] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 386996.000000 [08:31:39] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:32:55] MySQL slave on z-dat-s7-a is OK: Uptime: 51275 Threads: 10 Questions: 34534908 Slow queries: 3449 Opens: 55802 Flush tables: 1 Open tables: 3140 Queries per second avg: 673.523 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1731 [08:34:57] MySQL slave on z-dat-s3-a is OK: Uptime: 51389 Threads: 25 Questions: 55041632 Slow queries: 5560 Opens: 367882 Flush tables: 1 Open tables: 16384 Queries per second avg: 1071.78 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1756 [08:34:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:35:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:05] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 416768.000000 [08:58:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:21:05] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:57] aliasd on mayapple is CRITICAL: No data received from host [09:27:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 26031 MB (2% inode=99%): [09:28:08] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 67422 [09:28:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:28:56] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 68925 [09:28:56] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:28:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:29:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 70236 [09:29:58] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:29:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:29:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:29:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:29:58] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:30:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:05] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:45] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:30:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 262927.000000 [09:30:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:30:45] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 360907 MB (6% inode=63%): [09:31:39] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:31:39] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 407419.000000 [09:31:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:31:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65706.000000 [09:31:39] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70074.000000 [09:31:39] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 408804.000000 [09:31:39] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 489621.000000 [09:31:40] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45597 MB (7% inode=99%): [09:31:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 387686.000000 [09:31:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:31:41] / on thyme is UNKNOWN: NRPE: Unable to read output [09:31:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:34:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:35:05] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:05] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 418474.000000 [09:58:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:21:14] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:05] aliasd on mayapple is CRITICAL: No data received from host [10:28:08] Marlen Caemmerer * Re: [Toolserver-announce] [Toolserver-l] Failed Maintenance - was - Re: IP Renumbering of the complete Toolserver cluster [10:28:08] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63994 [10:28:14] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:28:56] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62743 [10:28:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:28:56] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:29:38] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92431 MB (15% inode=99%): [10:29:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62594 [10:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:30:14] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:14] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:45] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:30:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:30:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 360541 MB (6% inode=63%): [10:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:30:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:31:45] / on thyme is UNKNOWN: NRPE: Unable to read output [10:31:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:31:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 263142.000000 [10:32:37] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:32:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 408778.000000 [10:32:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69367.000000 [10:32:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:32:38] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62210.000000 [10:32:38] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 409414.000000 [10:32:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 491804.000000 [10:32:39] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:32:39] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 387327.000000 [10:34:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:35:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:15] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 420438.000000 [10:40:05] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [11:06:12] @replag [11:06:13] Alchimista: s1-rr-a: 16h 8m 31s [-2.11 s/s]; s1-rr-a-wd: 3d 22h 19m 18s [-0.07 s/s]; s1-user: 19h 49m 51s [+1.00 s/s]; s1-user-wd: 5d 16h 59m 34s [+0.60 s/s]; s2-rr: 16h 22m 28s [-1.77 s/s]; s2-user: 16h 22m 28s [-1.77 s/s]; s2-user-c: error; s2-user-wd: 4d 21h 2m 10s [+0.54 s/s] [11:06:14] Alchimista: s3-user: 6m 21s [+0.05 s/s]; s3-user-wd: 4d 11h 55m 25s [+0.08 s/s]; s4-user-wd: 4d 17h 55m 6s [+0.65 s/s]; s5-rr-a-wd: 3d 1h 7m 30s [+0.06 s/s]; s5-user: 14h 57m 56s [-3.72 s/s]; s6-user-wd: 4d 11h 34m 52s [-0.08 s/s]; s7-user-wd: 4d 17h 45m 27s [+0.11 s/s] [11:22:05] aliasd on mayapple is CRITICAL: No data received from host [11:27:06] aliasd on mayapple is OK: TCP OK - 0.001 second response time on port 984 [500 Not found.] [11:28:10] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57035 [11:28:13] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:28:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:29:38] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84330 MB (13% inode=99%): [11:29:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44899 [11:29:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:29:57] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55686 [11:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:29:58] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:30:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:15] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 72765 [11:30:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:30:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 360152 MB (6% inode=63%): [11:30:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:30:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:31:45] / on thyme is UNKNOWN: NRPE: Unable to read output [11:31:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:31:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 263359.000000 [11:32:37] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:32:37] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 411303.000000 [11:32:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72799.000000 [11:33:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:33:38] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42300.000000 [11:33:38] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 409694.000000 [11:33:38] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 494101.000000 [11:33:38] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 387714.000000 [11:33:38] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:35:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:36:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 422427.000000 [12:28:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52152 [12:28:14] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:28:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:29:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11592 [12:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:29:58] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48004 [12:29:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:29:58] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:30:15] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:15] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84092 MB (13% inode=99%): [12:31:46] / on thyme is UNKNOWN: NRPE: Unable to read output [12:31:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73618 [12:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:31:58] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359703 MB (6% inode=63%): [12:31:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:31:58] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:32:38] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:32:38] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 413331.000000 [12:32:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73617.000000 [12:32:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:32:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 263416.000000 [12:33:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:33:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 410140.000000 [12:33:42] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9784.000000 [12:33:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 496003.000000 [12:33:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:33:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 387996.000000 [12:35:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:38:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 424776.000000 [12:47:38] Free Memory on turnera is WARNING: WARNING - 6.9% (575392 kB) free! [12:48:37] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3457.000000 [12:48:38] Free Memory on turnera is OK: OK - 7.3% (609476 kB) free. [12:48:58] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3240 [12:56:38] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1787.000000 [12:56:57] MySQL slave on z-dat-s5-b is OK: Uptime: 20878 Threads: 8 Questions: 105152987 Slow queries: 226 Opens: 1874 Flush tables: 1 Open tables: 256 Queries per second avg: 5036.545 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1674 [12:57:37] Free Memory on turnera is WARNING: WARNING - 5.8% (481944 kB) free! [13:08:27] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:08:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:12:47] aliasd on yarrow is CRITICAL: Connection refused [13:13:26] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:28:06] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43506 [13:28:16] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:29:57] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37969 [13:29:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:29:58] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:30:16] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83796 MB (13% inode=99%): [13:31:46] / on thyme is UNKNOWN: NRPE: Unable to read output [13:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73637 [13:31:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359310 MB (6% inode=63%): [13:31:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 264724.000000 [13:32:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 415937.000000 [13:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73620.000000 [13:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:33:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 412299.000000 [13:34:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 498565.000000 [13:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:34:27] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 389651.000000 [13:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:35:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:16] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 427138.000000 [13:41:47] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2205.000000 [13:42:07] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2220.000000 [14:05:46] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [14:06:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3659.000000 [14:12:46] aliasd on yarrow is CRITICAL: Connection refused [14:15:32] @replag [14:15:33] Alchimista: s1-rr-a: 6h 29m 35s [-]; s1-rr-a-wd: 3d 23h 30m 49s [-]; s1-user: 20h 26m 5s [-]; s1-user-wd: 5d 18h 59m 1s [-]; s2-rr: 9h 57m 28s [-]; s2-user: 9h 57m 28s [-]; s2-user-c: error; s2-user-wd: 4d 23h 5m 35s [-] [14:15:34] Alchimista: s3-user: 1m 29s [-]; s3-user-wd: 4d 13h 13m 53s [-]; s4-user-wd: 4d 19h 53m 10s [-]; s5-rr-a-wd: 3d 2h 14m 57s [-]; s5-user: 22m 13s [-]; s5-user-c: 1h 10m 34s [-]; s6-user-wd: 4d 12h 55m 26s [-]; s7-user-wd: 4d 19h 13m 25s [-] [14:28:07] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32919 [14:28:17] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:29:57] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20637 [14:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:29:57] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:30:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:16] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83549 MB (13% inode=99%): [14:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [14:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73268 [14:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359249 MB (6% inode=63%): [14:31:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 268323.000000 [14:32:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 417819.000000 [14:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73310.000000 [14:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:33:59] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 415898.000000 [14:34:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 500844.000000 [14:34:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:34:27] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 393251.000000 [14:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:35:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 429709.000000 [15:05:46] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7245.000000 [15:06:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7259.000000 [15:12:47] aliasd on yarrow is CRITICAL: Connection refused [15:28:07] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25181 [15:28:17] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:29:57] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13048 [15:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:29:57] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:30:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83309 MB (13% inode=99%): [15:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [15:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73971 [15:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 358669 MB (6% inode=63%): [15:31:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 271924.000000 [15:32:48] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 420373.000000 [15:32:58] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:32:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73974.000000 [15:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:34:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 503125.000000 [15:34:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:34:27] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 396850.000000 [15:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 419558.000000 [15:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 431419.000000 [15:46:56] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 30265 MB (5% inode=99%): [16:05:47] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10845.000000 [16:06:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10859.000000 [16:12:47] aliasd on yarrow is CRITICAL: Connection refused [16:28:06] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18801 [16:28:16] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:29:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:29:57] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7524 [16:29:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:29:58] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:30:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83051 MB (13% inode=99%): [16:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 74255 [16:31:48] / on thyme is UNKNOWN: NRPE: Unable to read output [16:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 358139 MB (6% inode=63%): [16:31:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 275523.000000 [16:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:32:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 422734.000000 [16:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:33:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74198.000000 [16:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:34:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 504604.000000 [16:34:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:34:27] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 400451.000000 [16:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 423159.000000 [16:35:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 433407.000000 [16:55:14] @replag [16:55:15] Alchimista: s1-rr-a: 1h 29m 49s [-1.64 s/s]; s1-rr-a-wd: 4d 2h 10m 30s [+1.00 s/s]; s1-user: 20h 36m 45s [+0.12 s/s]; s1-user-wd: 5d 20h 14m 46s [+0.46 s/s]; s2-rr: 3h 47m 13s [-2.32 s/s]; s2-user: 3h 47m 13s [-2.32 s/s]; s2-user-c: error; s2-user-wd: 5d 27m 47s [+0.48 s/s] [16:55:16] Alchimista: s3-user: 25s [+0.00 s/s]; s3-user-wd: 4d 15h 53m 34s [+1.00 s/s]; s4-user-wd: 4d 21h 43m 22s [+0.70 s/s]; s5-rr-a-wd: 3d 4h 54m 38s [+1.00 s/s]; s5-user-c: 3h 50m 15s [+1.00 s/s]; s6-user-wd: 4d 15h 35m 7s [+1.00 s/s]; s7-user-wd: 4d 21h 53m 6s [+1.00 s/s] [17:05:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14447.000000 [17:06:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14459.000000 [17:12:47] aliasd on yarrow is CRITICAL: Connection refused [17:15:57] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3517 [17:28:06] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8466 [17:28:16] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:29:57] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:30:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82805 MB (13% inode=99%): [17:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73579 [17:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [17:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 357749 MB (6% inode=63%): [17:31:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 279123.000000 [17:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:32:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 425553.000000 [17:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:32:57] MySQL slave on z-dat-s1-b is OK: Uptime: 37474 Threads: 11 Questions: 47299297 Slow queries: 507 Opens: 563 Flush tables: 1 Open tables: 372 Queries per second avg: 1262.189 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1652 [17:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73658.000000 [17:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:34:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 505019.000000 [17:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:34:27] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 404050.000000 [17:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 426759.000000 [17:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 434366.000000 [17:47:36] @replag [17:47:37] Betacommand: s1-rr-a-wd: 4d 3h 2m 52s [+1.00 s/s]; s1-user: 20h 26m 9s [-0.20 s/s]; s1-user-wd: 5d 20h 17m 15s [+0.05 s/s]; s2-rr: 1h 15m 21s [-2.90 s/s]; s2-user: 1h 15m 21s [-2.90 s/s]; s2-user-c: error; s2-user-wd: 5d 45m 24s [+0.34 s/s]; s3-user: 28s [+0.00 s/s] [17:47:37] Betacommand: s3-user-wd: 4d 16h 45m 56s [+1.00 s/s]; s4-user-wd: 4d 22h 22m 0s [+0.74 s/s]; s5-rr-a-wd: 3d 5h 47m 0s [+1.00 s/s]; s5-user: 1m 16s [-0.14 s/s]; s5-user-c: 4h 42m 37s [+1.00 s/s]; s6-user-wd: 4d 16h 27m 29s [+1.00 s/s]; s7-user-wd: 4d 22h 45m 28s [+1.00 s/s] [17:51:06] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3504 [18:05:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18047.000000 [18:06:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18060.000000 [18:07:06] MySQL slave on z-dat-s2-b is OK: Uptime: 39507 Threads: 15 Questions: 57076328 Slow queries: 691 Opens: 539948 Flush tables: 1 Open tables: 256 Queries per second avg: 1444.714 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1697 [18:12:47] aliasd on yarrow is CRITICAL: Connection refused [18:28:16] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:29:56] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:29:57] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:30:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82528 MB (13% inode=99%): [18:31:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 72743 [18:31:46] / on thyme is UNKNOWN: NRPE: Unable to read output [18:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:31:58] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 357330 MB (6% inode=63%): [18:31:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 282723.000000 [18:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:32:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 427515.000000 [18:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:33:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72691.000000 [18:33:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:34:19] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 505320.000000 [18:34:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:34:26] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 407651.000000 [18:34:26] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 430359.000000 [18:35:16] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 436097.000000 [19:05:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21648.000000 [19:06:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21660.000000 [19:12:47] aliasd on yarrow is CRITICAL: Connection refused [19:19:01] Free Memory on damiana is WARNING: WARNING - 5.9% (498444 kB) free! [19:19:57] Free Memory on damiana is OK: OK - 8.2% (686860 kB) free. [19:27:57] Free Memory on damiana is CRITICAL: CRITICAL - 3.1% (261504 kB) free! [19:28:16] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:29:58] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:29:58] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:29:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:29:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:29:58] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:30:16] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82188 MB (13% inode=99%): [19:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 70759 [19:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [19:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356936 MB (6% inode=63%): [19:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 286323.000000 [19:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [19:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70838.000000 [19:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 429929.000000 [19:34:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506633.000000 [19:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:34:27] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 411250.000000 [19:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 433959.000000 [19:35:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:16] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 436527.000000 [19:55:57] Free Memory on damiana is CRITICAL: CRITICAL - 2.3% (193136 kB) free! [20:05:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25248.000000 [20:06:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25259.000000 [20:12:47] aliasd on yarrow is CRITICAL: Connection refused [20:28:17] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:29:57] Free Memory on damiana is WARNING: WARNING - 5.2% (433676 kB) free! [20:29:58] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:30:18] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:30:18] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81944 MB (13% inode=99%): [20:30:56] Free Memory on damiana is CRITICAL: CRITICAL - 3.2% (269180 kB) free! [20:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 70127 [20:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [20:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356669 MB (6% inode=63%): [20:32:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 289924.000000 [20:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70207.000000 [20:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 431895.000000 [20:34:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506910.000000 [20:34:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:34:27] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 414850.000000 [20:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 437558.000000 [20:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 437677.000000 [20:39:58] Free Memory on damiana is WARNING: WARNING - 5.1% (431108 kB) free! [20:59:57] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (392936 kB) free! [21:05:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28848.000000 [21:06:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28859.000000 [21:12:47] aliasd on yarrow is CRITICAL: Connection refused [21:27:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 25575 MB (2% inode=99%): [21:28:16] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:28:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:29:57] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81657 MB (13% inode=99%): [21:31:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 69958 [21:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [21:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356366 MB (6% inode=63%): [21:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 293524.000000 [21:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69929.000000 [21:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 433681.000000 [21:34:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506956.000000 [21:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:34:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 418460.000000 [21:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 441159.000000 [21:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 438345.000000 [21:59:57] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (370392 kB) free! [22:05:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32448.000000 [22:06:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32459.000000 [22:12:47] aliasd on yarrow is CRITICAL: Connection refused [22:19:57] Free Memory on damiana is WARNING: WARNING - 5.1% (423620 kB) free! [22:21:57] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (364296 kB) free! [22:28:16] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:30:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81412 MB (13% inode=99%): [22:30:57] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:31:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 66270 [22:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [22:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 355534 MB (6% inode=63%): [22:32:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 297124.000000 [22:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66118.000000 [22:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 436038.000000 [22:34:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506336.000000 [22:34:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:34:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 422059.000000 [22:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 444758.000000 [22:34:57] Free Memory on damiana is WARNING: WARNING - 5.4% (448584 kB) free! [22:35:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 440379.000000 [22:40:57] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (367296 kB) free! [22:45:09] @replag [22:45:11] Betacommand: s1-rr-a: 6m 22s [-0.24 s/s]; s1-rr-a-wd: 4d 8h 26s [+1.00 s/s]; s1-user: 18h 3m 11s [-0.48 s/s]; s1-user-wd: 5d 20h 37m 45s [+0.07 s/s]; s2-user-c: error; s2-user-wd: 5d 2h 19m 53s [+0.32 s/s]; s3-user: 15s [-0.00 s/s]; s3-user-wd: 4d 21h 43m 30s [+1.00 s/s] [22:45:11] Betacommand: s4-user-wd: 5d 1h 9m 54s [+0.56 s/s]; s5-rr-a-wd: 3d 10h 44m 34s [+1.00 s/s]; s5-user-c: 9h 40m 11s [+1.00 s/s]; s6-user: 39s [-]; s6-user-wd: 4d 21h 25m 3s [+1.00 s/s]; s7-user-wd: 5d 3h 43m 2s [+1.00 s/s] [22:50:56] Free Memory on damiana is WARNING: WARNING - 5.1% (426848 kB) free! [23:05:57] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27105.000000 [23:06:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30054.000000 [23:10:56] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422132 kB) free! [23:12:46] aliasd on yarrow is CRITICAL: Connection refused [23:28:17] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:29:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:30:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81172 MB (13% inode=99%): [23:31:01] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:31:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62718 [23:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [23:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 354454 MB (6% inode=63%): [23:32:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 297643.000000 [23:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:32:56] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62724.000000 [23:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 436835.000000 [23:34:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 506948.000000 [23:34:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:34:26] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:34:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 423045.000000 [23:34:56] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 446300.000000 [23:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:36:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 440676.000000 [23:39:58] Free Memory on damiana is WARNING: WARNING - 5.2% (434244 kB) free! [23:41:59] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (360844 kB) free! [23:50:56] Free Memory on damiana is WARNING: WARNING - 5.3% (446796 kB) free! [23:55:56] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3275.000000 [23:58:57] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 902.000000