[00:02:33] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:02:43] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:16:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3654.000000 [00:22:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [00:31:23] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2207.000000 [00:31:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:31:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:32:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 175735 MB (3% inode=68%): [00:32:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 96370.000000 [00:32:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:32:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:32:23] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:32:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:32:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:32:33] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 672312 [00:32:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84514 MB (13% inode=99%): [00:32:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:53] aliasd on mayapple is CRITICAL: Connection refused [00:32:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:33:28] nacht ts [00:33:53] @replag [00:33:54] DaBPunkt: s1-rr-a-wd: 32s [-]; s1-user: 1m 17s [-]; s1-user-c: 1h 18m 23s [-]; s1-user-wd: 32s [-]; s2-rr: 39m 29s [-]; s2-user: 39m 29s [-]; s2-user-c: error; s2-user-wd: 1d 2h 40m 28s [-] [00:33:55] DaBPunkt: s3-user: 26s [-]; s3-user-wd: 30s [-]; s4-rr-a: 11m 58s [-]; s4-user: 11m 58s [-]; s4-user-wd: 1m 29s [-]; s5-user: 39m 30s [-]; s5-user-c: error; s6-user: 4m 35s [-] [00:33:56] DaBPunkt: s6-user-wd: 27m 3s [-]; s7-user-wd: 32s [-] [00:34:20] @replag [00:34:20] DaBPunkt: s1-rr-a-wd: 33s [+0.04 s/s]; s1-user: 1m 24s [+0.27 s/s]; s1-user-c: 1h 18m 49s [+0.99 s/s]; s1-user-wd: 37s [+0.19 s/s]; s2-rr: 39m 55s [+0.99 s/s]; s2-user: 39m 55s [+0.99 s/s]; s2-user-c: error; s2-user-wd: 1d 2h 39m 6s [-3.13 s/s] [00:34:21] DaBPunkt: s3-user: 11s [-0.57 s/s]; s3-user-wd: 33s [+0.11 s/s]; s4-rr-a: 11m 23s [-1.34 s/s]; s4-user: 11m 23s [-1.34 s/s]; s4-user-wd: 1m 27s [-0.08 s/s]; s5-user: 39m 56s [+0.99 s/s]; s5-user-c: error; s6-user: 4m 39s [+0.15 s/s] [00:34:22] DaBPunkt: s6-user-wd: 27m 9s [+0.23 s/s]; s7-user-wd: 35s [+0.12 s/s] [00:40:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [00:46:24] MySQL on z-dat-s5-b is CRITICAL: Cant connect to MySQL server on z-dat-s5-b (146) [00:46:34] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [00:47:03] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2177.000000 [00:54:03] wikidata replag on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [00:54:33] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2538.000000 [00:56:24] MySQL on z-dat-s5-b is OK: Uptime: 19 Threads: 3 Questions: 268 Slow queries: 0 Opens: 55 Flush tables: 1 Open tables: 44 Queries per second avg: 14.105 [00:56:34] MySQL on z-dat-s2-b is OK: Uptime: 30 Threads: 12 Questions: 447 Slow queries: 1 Opens: 3909 Flush tables: 1 Open tables: 132 Queries per second avg: 14.900 [00:58:43] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3557 [00:59:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3446.000000 [01:02:43] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3975 [01:06:11] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 61175 MB (6% inode=99%): [01:06:21] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:06:31] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1482.000000 [01:07:21] MySQL slave on z-dat-s5-b is OK: Uptime: 668 Threads: 2 Questions: 402258 Slow queries: 5 Opens: 181 Flush tables: 1 Open tables: 144 Queries per second avg: 602.182 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 814 [01:09:01] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3496 [01:11:54] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1.000000 [01:14:44] is there some backup of my account ? [01:15:02] these evening i have lost a file [01:15:18] while i transfer a new version [01:17:24] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7284.000000 [01:21:34] MySQL slave on z-dat-s2-b is OK: Uptime: 1530 Threads: 9 Questions: 691573 Slow queries: 25 Opens: 26211 Flush tables: 1 Open tables: 256 Queries per second avg: 452.8 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1719 [01:23:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5106.000000 [01:31:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94503.000000 [01:32:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:32:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:32:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:32:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:32:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:32:55] aliasd on mayapple is CRITICAL: Connection refused [01:32:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 175712 MB (3% inode=68%): [01:33:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:33:14] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:33:14] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:33:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:33:24] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 643049 [01:33:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84066 MB (13% inode=99%): [01:33:44] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:24] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [02:03:25] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [02:10:07] scfc_de: https://en.wikipedia.org/wiki/User_talk:MZMcBride#Please_investigate_broken_report. [02:10:28] Oh, I'm still running that one. [02:10:33] I wonder why it's broken. [02:11:39] Hmm, the ORDER BY. [02:17:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6785.000000 [02:18:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4013.000000 [02:20:24] MySQL slave on z-dat-s3-a is OK: Uptime: 1259 Threads: 19 Questions: 1834 Slow queries: 0 Opens: 233 Flush tables: 1 Open tables: 222 Queries per second avg: 1.456 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1427 [02:20:25] MySQL on z-dat-s3-a is OK: Uptime: 1259 Threads: 20 Questions: 1837 Slow queries: 0 Opens: 233 Flush tables: 1 Open tables: 222 Queries per second avg: 1.459 [02:23:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4495.000000 [02:26:44] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3590.000000 [02:31:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86328.000000 [02:32:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:32:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:32:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:32:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:32:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:32:55] aliasd on mayapple is CRITICAL: Connection refused [02:32:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 174294 MB (3% inode=68%): [02:33:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:33:14] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:33:14] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:33:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:33:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84267 MB (13% inode=99%): [02:33:44] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 613745 [02:33:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:34:14] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (127868 kB) free! [02:35:50] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:36:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:38:20] Free Memory on damiana is OK: OK - 88.8% (7437556 kB) free. [02:40:20] Susan: As it's broken, I haven't even transferred it to SGE, so your responsibility alone :-) (or you need to wait some time :-)). [02:58:55] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2079 [02:58:56] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2082.000000 [03:06:55] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1754.000000 [03:07:55] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3577.000000 [03:07:55] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1781.000000 [03:08:55] MySQL slave on z-dat-s5-b is OK: Uptime: 7959 Threads: 7 Questions: 1478412 Slow queries: 118 Opens: 1216 Flush tables: 1 Open tables: 246 Queries per second avg: 185.753 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1571 [03:19:00] Free Memory on turnera is CRITICAL: CRITICAL - 1.7% (142740 kB) free! [03:24:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4980.000000 [03:27:05] Free Memory on turnera is OK: OK - 86.5% (7246348 kB) free. [03:32:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83386.000000 [03:32:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:32:35] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2401.000000 [03:32:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:32:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:33:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:33:06] aliasd on mayapple is CRITICAL: Connection refused [03:33:06] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 173984 MB (3% inode=68%): [03:33:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:33:25] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:33:26] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:33:35] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1773.000000 [03:33:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83816 MB (13% inode=99%): [03:33:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:34:35] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 578852 [03:36:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:54:35] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [04:02:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:10:55] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2194.000000 [04:26:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6581.000000 [04:32:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80398.000000 [04:32:30] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:32:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:33:13] aliasd on mayapple is CRITICAL: Connection refused [04:33:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:33:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:33:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 173598 MB (3% inode=68%): [04:33:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:33:54] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:34:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83592 MB (13% inode=99%): [04:34:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:34:30] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:54] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 566640 [04:37:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:11:29] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3109.000000 [20:38:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:38:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:38:25] aliasd on mayapple is CRITICAL: Connection refused [20:38:25] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:38:36] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 163410 MB (3% inode=68%): [20:39:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:39:01] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:39:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:39:07] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 65748 [20:39:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:39:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:40:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:45:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2008.000000 [20:45:43] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2026 [20:54:06] Merlissimo, I've the same trouble again on yarrow, job 1480476, when this job is scheduled on nightshade I've no trouble to qdel it [20:58:51] ha, qdel worked [21:11:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:28:05] DaB. * [Toolserver-announce] Introduction: Operators (was: Adding more roots) [21:33:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25665.000000 [21:35:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78797 MB (12% inode=99%): [21:36:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:04] aliasd on mayapple is CRITICAL: Connection refused [21:36:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:36:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 163046 MB (3% inode=68%): [21:36:34] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:36:34] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:36:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:36:44] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31210 [21:37:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:37:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:39:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:45:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3045.000000 [21:45:44] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3045 [21:57:54] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [22:11:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:23:54] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [22:33:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20645.000000 [22:35:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3617.000000 [22:35:44] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3633 [22:35:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78428 MB (12% inode=99%): [22:36:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:36:05] aliasd on mayapple is CRITICAL: Connection refused [22:36:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:36:15] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 162631 MB (3% inode=68%): [22:36:34] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:36:34] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:36:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:36:44] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13318 [22:37:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:37:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:39:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:47:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3571.000000 [22:47:44] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3539 [23:02:44] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3528 [23:10:44] MySQL slave on z-dat-s4-a is OK: Uptime: 4060701 Threads: 1 Questions: 421247666 Slow queries: 233 Opens: 667 Flush tables: 11 Open tables: 222 Queries per second avg: 103.737 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1799 [23:11:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:33:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:34:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14065.000000 [23:35:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78137 MB (12% inode=99%): [23:36:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:36:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:36:04] aliasd on mayapple is CRITICAL: Connection refused [23:36:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:36:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 160774 MB (3% inode=68%): [23:36:34] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:36:34] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:36:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:37:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:37:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:39:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:47:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3051.000000 [23:47:48] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3066