[00:00:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:00:48] DaBPunkt: thanks. isn't possible to do it on mysql console? I'm getting a ERROR 1045 (28000): Access denied for user 'alchimista'@'%.toolserver.org' (using password: YES) [00:00:57] but that should work, anyway [00:01:18] mysql -h sql-s1 -B enwiki_p -e "select max(rc_id) from recentchanges;">test.txt [00:01:33] Alchimista: no, not possible [00:01:54] ok :D [00:02:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:04:46] @replag [00:04:46] DaBPunkt: s1-rr-a-wd: 21h 30m 5s [-6.52 s/s]; s1-user-wd: 22h 6m 40s [-6.72 s/s]; s2-user-wd: 3d 5h 35m 50s [+1.00 s/s]; s3-rr-a: 42s [-0.02 s/s]; s3-user: 42s [-0.02 s/s]; s4-rr-a: 9m 23s [-1.43 s/s]; s4-user: 9m 23s [-1.43 s/s]; s4-user-wd: 3d 2h 47m 0s [-2.38 s/s] [00:04:46] DaBPunkt: s5-user-wd: 3d 5h 35m 51s [+1.00 s/s] [00:05:43] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [00:07:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:11:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:21:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:32:04] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2190.000000 [00:49:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [00:49:15] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:49:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:50:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:50:04] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:51:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:54:04] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1320 MB (9% inode=99%): [00:56:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3631.000000 [00:56:32] @replag [00:56:32] DaBPunkt: s1-rr-a-wd: 15h 26m 50s [-7.01 s/s]; s1-user-wd: 16h 2m 34s [-7.03 s/s]; s2-user-wd: 3d 6h 27m 38s [+1.00 s/s]; s3-rr-a: 10s [-0.01 s/s]; s3-user: 10s [-0.01 s/s]; s4-rr-a: 1h 1m 11s [+1.00 s/s]; s4-user: 1h 1m 11s [+1.00 s/s]; s4-user-wd: 3d 2h 7m 24s [-0.76 s/s] [00:56:33] DaBPunkt: s5-user-wd: 3d 6h 27m 39s [+1.00 s/s] [00:57:06] /tmp on ortelius is OK: DISK OK - free space: /tmp 13016 MB (98% inode=99%): [00:57:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:57:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:57:52] nacht ts [01:00:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:03:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:05:46] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [01:07:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:11:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:21:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:49:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:50:04] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:50:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:50:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [01:50:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:52:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:56:05] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7230.000000 [01:57:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:58:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:00:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:03:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:05:44] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [02:07:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:12:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:21:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:49:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:50:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:50:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:50:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [02:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:52:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:56:12] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10830.000000 [02:57:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:58:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:00:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:03:39] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:05:44] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [03:07:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:12:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:21:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:49:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:50:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:50:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:50:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [03:50:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:52:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:56:07] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14431.000000 [03:58:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:58:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:00:46] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:04:02] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:06:01] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [04:07:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:12:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:21:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:49:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:50:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:50:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:50:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:52:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:56:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18091.000000 [04:58:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:58:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:01:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:04:57] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:06:58] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [05:07:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:13:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:21:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:49:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:50:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:50:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:50:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:50:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:52:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:57:57] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21751.000000 [05:58:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:59:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:01:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:04:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:06:58] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [06:07:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:13:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:49:42] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:50:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:50:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:50:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:50:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:51:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:52:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:57:57] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25350.000000 [06:58:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:59:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:01:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:04:57] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:06:58] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [07:07:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:12:57] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 127503 MB (13% inode=99%): [07:13:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:49:39] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:50:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [07:50:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:50:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:50:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:52:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:57:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28950.000000 [07:58:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:59:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:01:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:03:17] /sql on thyme is WARNING: DISK WARNING - free space: /sql 197190 MB (20% inode=99%): [08:04:57] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:07:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:08:01] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [08:14:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:17:12] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58509 MB (5% inode=97%): [08:17:17] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58381 MB (5% inode=97%): [08:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:22:11] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59438 MB (6% inode=97%): [08:22:18] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59438 MB (6% inode=97%): [08:30:57] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 27778 MB (6% inode=99%): [08:42:30] Load avg. on adenia is WARNING: WARNING - load average: 15.74, 17.58, 14.07 [08:45:27] Load avg. on adenia is OK: OK - load average: 9.09, 14.05, 13.33 [08:50:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [08:50:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:50:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:38] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:52:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:54:27] Load avg. on adenia is WARNING: WARNING - load average: 19.99, 18.90, 16.00 [08:58:01] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32550.000000 [08:58:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:59:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:01:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:04:59] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:07:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:07:59] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [09:14:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:21:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:50:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:50:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:51:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [09:53:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:57:58] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36151.000000 [09:59:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:00:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:01:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:04:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:07:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:07:57] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [10:14:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:34:09] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58456 MB (5% inode=97%): [10:35:08] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 60440 MB (6% inode=97%): [10:50:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:50:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:50:29] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:50:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:51:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [10:53:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:58:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39810.000000 [10:59:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:00:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:04:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:07:58] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [11:08:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:14:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:17:13] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58362 MB (5% inode=97%): [11:17:18] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58358 MB (5% inode=97%): [11:19:08] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58622 MB (6% inode=97%): [11:20:09] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58353 MB (5% inode=97%): [11:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:22:18] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58615 MB (6% inode=97%): [11:23:18] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58269 MB (5% inode=97%): [11:43:23] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59764 MB (6% inode=97%): [11:44:10] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59789 MB (6% inode=97%): [11:50:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:51:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [11:51:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:51:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:51:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:53:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:58:57] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43410.000000 [11:59:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:00:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:01:57] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2175 [12:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:02:47] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2155 [12:05:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:08:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:08:57] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [12:09:58] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2191 [12:14:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:21:32] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:26:29] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:26:59] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3645 [12:26:59] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 28193 MB (6% inode=99%): [12:28:37] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3657 [12:29:59] aliasd on yarrow is CRITICAL: Connection refused [12:37:59] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3631 [12:41:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 100770 MB (10% inode=99%): [12:50:38] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:51:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [12:51:26] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:51:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:51:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:53:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:58:58] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47010.000000 [12:59:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:00:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:05:57] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:08:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:10:00] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [13:15:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:27:00] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7151 [13:28:38] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7105 [13:29:58] MySQL slave on daphne is OK: Uptime: 48671 Threads: 35 Questions: 8400054 Slow queries: 522 Opens: 2937 Flush tables: 1 Open tables: 942 Queries per second avg: 172.588 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 687 [13:29:58] aliasd on yarrow is CRITICAL: Connection refused [13:29:58] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 688.000000 [13:37:57] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6920 [13:40:47] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:41:18] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59484 MB (6% inode=97%): [13:50:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:51:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:51:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:51:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:51:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:53:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:53:57] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 127874 MB (13% inode=99%): [13:59:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:00:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:05:57] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:08:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:15:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:21:27] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:21:57] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 28116 MB (6% inode=99%): [14:27:28] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:27:38] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10609 [14:28:47] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7527 [14:30:00] aliasd on yarrow is CRITICAL: Connection refused [14:37:58] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10240 [14:46:57] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 28099 MB (6% inode=99%): [14:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:51:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:51:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:51:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:51:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:52:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [14:54:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:54:59] aliasd on yarrow is OK: TCP OK - 0.014 second response time on port 984 [500 Not found.] [14:55:28] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:55:37] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:55:47] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:56:13] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 28095 MB (6% inode=99%): [14:56:13] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59297 MB (6% inode=97%): [14:56:17] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59238 MB (6% inode=97%): [15:00:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:01:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:05:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:08:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:14:59] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3487 [15:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:17:59] MySQL slave on z-dat-s7-a is OK: Uptime: 111679 Threads: 6 Questions: 34878123 Slow queries: 6021 Opens: 330304 Flush tables: 1 Open tables: 5058 Queries per second avg: 312.306 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1482 [15:19:00] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3540 [15:27:02] MySQL slave on z-dat-s6-a is OK: Uptime: 624504 Threads: 5 Questions: 122770154 Slow queries: 28374 Opens: 1005952 Flush tables: 1 Open tables: 2956 Queries per second avg: 196.588 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1667 [15:28:09] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9080 [15:46:08] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3327 [15:51:34] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:51:34] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:51:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:51:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:51:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:52:09] MySQL slave on z-dat-s3-a is OK: Uptime: 113848 Threads: 14 Questions: 104528069 Slow queries: 23412 Opens: 1245129 Flush tables: 1 Open tables: 16384 Queries per second avg: 918.137 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1747 [15:52:28] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [15:55:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:00:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:00:33] @replag [16:00:33] Brooke: s1-rr-a-wd: 3h 39m 39s [+0.54 s/s]; s1-user-wd: 4h 11m 52s [+0.61 s/s]; s2-user: 11s [+0.00 s/s]; s2-user-wd: 3d 19h 20m 47s [-0.72 s/s]; s3-rr-a: 21s [-2.48 s/s]; s3-user: 21s [-2.48 s/s]; s4-user-wd: 3d 16h 20m 3s [+0.32 s/s]; s5-user-wd: 3d 19h 20m 47s [-0.73 s/s] [16:01:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:02:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:06:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:08:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:26:49] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:31:37] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:51:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:51:29] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:51:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:52:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [16:52:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:55:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:00:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:01:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:02:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:07:00] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:08:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:52:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:52:29] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [17:52:29] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:52:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:52:39] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:55:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:58:08] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58475 MB (5% inode=97%): [17:58:16] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58459 MB (5% inode=97%): [18:01:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:01:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:02:08] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59268 MB (6% inode=97%): [18:02:20] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59195 MB (6% inode=97%): [18:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:07:00] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:08:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:52:31] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [18:52:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:52:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:52:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:52:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:55:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:01:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:01:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:06:12] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58428 MB (5% inode=97%): [19:06:16] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58375 MB (5% inode=97%): [19:06:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:08:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:12:09] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59434 MB (6% inode=97%): [19:12:18] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59407 MB (6% inode=97%): [19:15:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:21:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:52:28] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [19:52:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:52:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:52:29] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:52:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:56:16] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58491 MB (5% inode=97%): [19:56:41] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:57:16] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58580 MB (6% inode=97%): [20:01:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:02:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:02:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:03:33] /sql on thyme is WARNING: DISK WARNING - free space: /sql 197087 MB (20% inode=99%): [20:07:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:08:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:15:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:21:34] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:36:09] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58502 MB (5% inode=97%): [20:37:08] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58551 MB (6% inode=97%): [20:50:13] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58512 MB (5% inode=97%): [20:52:38] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:53:30] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [20:53:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:53:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:53:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:57:20] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58516 MB (5% inode=97%): [20:57:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:01:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:02:29] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:02:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:07:57] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:09:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:11:09] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58468 MB (5% inode=97%): [21:14:09] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58553 MB (6% inode=97%): [21:14:22] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58854 MB (6% inode=97%): [21:15:08] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58415 MB (5% inode=97%): [21:15:16] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58411 MB (5% inode=97%): [21:15:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:22:08] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58962 MB (6% inode=97%): [21:22:20] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58849 MB (6% inode=97%): [21:52:38] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:54:29] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [21:54:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:54:29] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:54:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:57:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:01:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:02:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:03:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:08:09] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:10:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:15:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:53:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:55:28] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [22:55:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:55:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:28] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:02:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:03:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:03:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:08:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:10:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:16:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:27:09] MZMcBride * [Toolserver-l] rosemary's copy of enwiki_p also corrupt? [23:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:53:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:55:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:55:29] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [23:55:29] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:55:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:57:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)