[00:02:04] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:05:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:05:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:05:49] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [00:07:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:09:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:11:46] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:13:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:13:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:16:33] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:22:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:24:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:51:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:02:09] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:05:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:50] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [01:07:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:09:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:11:46] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:13:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:13:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:14:30] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 27168 MB (6% inode=99%): [01:16:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:22:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:24:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:51:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:54:29] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 125924 MB (12% inode=99%): [02:02:09] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:05:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:05:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:06:45] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [02:08:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:09:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:12:45] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:13:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:13:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:16:26] Load avg. on ortelius is WARNING: WARNING - load average: 18.55, 16.45, 11.16 [02:17:36] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:22:35] Load avg. on ortelius is OK: OK - load average: 10.12, 14.89, 12.52 [02:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:24:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:51:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:02:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:05:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:05:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:06:45] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [03:08:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:09:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:12:45] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:13:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:13:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:17:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:25:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:51:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:02:08] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:05:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:05:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:06:58] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:08:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:09:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:12:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:13:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:13:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:17:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:23:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:25:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:51:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:56:29] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 56947 MB (5% inode=97%): [04:56:29] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 56945 MB (5% inode=97%): [05:03:07] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:06:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:06:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:06:58] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:08:19] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:10:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:12:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:13:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:13:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:17:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:24:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:26:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:51:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:04:07] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:06:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:06:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:06:57] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:08:22] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:11:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:12:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:13:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:17:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:25:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:26:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:51:33] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:05:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:06:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:06:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:07:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [07:09:20] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:11:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:13:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:13:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:13:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:18:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:25:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:25:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:04:58] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196339 MB (20% inode=99%): [08:05:09] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:06:09] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:06:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:08:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [08:09:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:11:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:13:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:13:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:13:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:18:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:25:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:26:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:58:58] Load avg. on adenia is WARNING: WARNING - load average: 17.16, 16.80, 14.76 [09:06:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:06:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:06:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:08:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [09:09:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:11:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:11:58] Load avg. on adenia is OK: OK - load average: 13.57, 14.77, 14.93 [09:13:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:13:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:14:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:18:39] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:18:57] Load avg. on adenia is WARNING: WARNING - load average: 13.50, 15.54, 15.42 [09:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:25:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:26:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:06:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:06:07] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:06:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:08:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [10:09:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:11:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:13:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:13:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:14:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:19:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:20:58] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2177 [10:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:25:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:26:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:29:27] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:29:39] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:29:39] MySQL slave on z-dat-s3-a is CRITICAL: slave query error: Lost connection to MySQL server during query [10:30:28] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 27414 MB (6% inode=99%): [10:30:28] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 56359 MB (5% inode=97%): [10:31:58] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2173 [10:35:17] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [10:35:18] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [10:36:17] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [10:37:17] MySQL on z-dat-s6-a is OK: Uptime: 458 Threads: 6 Questions: 7515 Slow queries: 6 Opens: 181 Flush tables: 1 Open tables: 151 Queries per second avg: 16.408 [10:37:17] MySQL slave on z-dat-s6-a is OK: Uptime: 458 Threads: 6 Questions: 7553 Slow queries: 6 Opens: 181 Flush tables: 1 Open tables: 151 Queries per second avg: 16.491 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1731 [10:38:02] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [10:42:05] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2632 [10:51:17] MySQL on z-dat-s3-a is OK: Uptime: 1296 Threads: 3 Questions: 21 Slow queries: 0 Opens: 26 Flush tables: 1 Open tables: 18 Queries per second avg: 0.16 [11:06:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:06:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:06:12] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:08:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [11:09:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:11:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:14:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:15:00] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:15:00] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3550 [11:15:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:20:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:24:58] MySQL slave on z-dat-s3-a is OK: Uptime: 3315 Threads: 17 Questions: 1034152 Slow queries: 36 Opens: 19576 Flush tables: 1 Open tables: 11894 Queries per second avg: 311.961 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1545 [11:26:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:27:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:43:27] hello all [11:43:33] @replag [11:43:33] DaBPunkt: s1-rr-a-wd: 30s [-0.00 s/s]; s1-user-wd: 6h 25m 56s [+0.27 s/s]; s2-user-wd: 14h 5m 17s [-1.07 s/s]; s3-rr-a: 1m 59s [+0.00 s/s]; s3-user: 1m 59s [+0.00 s/s]; s4-user-wd: 6h 26m 4s [-0.42 s/s]; s5-user-wd: 14h 5m 17s [-1.07 s/s] [11:43:56] mm, looks like there is still a problem somewhere :( [11:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:06:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:06:07] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:06:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:08:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [12:10:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:11:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:14:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:14:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:15:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:17:57] Free Memory on turnera is WARNING: WARNING - 5.2% (431656 kB) free! [12:21:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:21:57] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (416820 kB) free! [12:22:58] Free Memory on turnera is WARNING: WARNING - 5.2% (431968 kB) free! [12:26:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:27:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:51:33] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:56:02] Free Memory on turnera is WARNING: WARNING - 5.4% (455336 kB) free! [12:59:02] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (238024 kB) free! [13:02:58] Free Memory on turnera is WARNING: WARNING - 5.1% (423684 kB) free! [13:06:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:06:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:06:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:08:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:10:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:11:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:14:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:15:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:15:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:22:48] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:23:57] Free Memory on turnera is WARNING: WARNING - 5.2% (435480 kB) free! [13:26:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:27:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:51:26] Reedy: hi [13:51:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:51:56] do you know how I can gain access and upload to http://toolserver.org/~awb/snapshots/ ? [13:53:29] I am logged in https://wiki.toolserver.org as magioladitis [13:54:07] or alternatively I could get my own space to upload snapshots of awb? [13:54:29] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 125437 MB (12% inode=99%): [13:55:00] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420716 kB) free! [13:56:00] Free Memory on turnera is WARNING: WARNING - 5.3% (445844 kB) free! [13:59:00] Free Memory on turnera is CRITICAL: CRITICAL - 3.1% (262340 kB) free! [14:06:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:06:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:07:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:08:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [14:10:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:11:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:14:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:16:00] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:16:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:18:57] Free Memory on turnera is WARNING: WARNING - 5.2% (438948 kB) free! [14:23:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:25:49] if anyone can help, please ping me [14:26:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:27:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:39:57] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417220 kB) free! [14:41:07] Free Memory on turnera is WARNING: WARNING - 5.2% (436396 kB) free! [14:43:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422908 kB) free! [14:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:06:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:06:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:07:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:08:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [15:10:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:11:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:14:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:15:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:16:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:23:58] Free Memory on turnera is WARNING: WARNING - 5.2% (436864 kB) free! [15:24:48] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:24:59] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420940 kB) free! [15:25:58] Free Memory on turnera is WARNING: WARNING - 5.1% (423620 kB) free! [15:26:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:27:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:33:06] @replag [15:33:06] DaBPunkt: s1-rr-a-wd: 9m 42s [+0.04 s/s]; s1-user-wd: 9m 48s [-1.64 s/s]; s2-user-wd: 9m 48s [-3.64 s/s]; s3-rr-a: 10s [-0.01 s/s]; s3-user: 10s [-0.01 s/s]; s4-user-wd: 9m 44s [-1.64 s/s]; s5-user-wd: 9m 48s [-3.64 s/s] [15:39:59] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (421696 kB) free! [15:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:06:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:06:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:07:07] Daniel Schwen * Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt? [16:07:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:08:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [16:10:20] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:11:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:14:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:15:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:16:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:18:57] Free Memory on turnera is WARNING: WARNING - 6.2% (516136 kB) free! [16:25:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:27:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:27:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:44:35] @replag [16:44:35] liangent: s1-rr-a-wd: 19m 51s [+0.14 s/s]; s1-user-wd: 16m 22s [+0.09 s/s]; s2-user-wd: 20m 47s [+0.15 s/s]; s4-user-wd: 20m 46s [+0.15 s/s]; s5-user-wd: 20m 47s [+0.15 s/s] [16:47:03] I'm debugging the replication of wikidata, so it can stop from time to time [16:48:41] @replag [16:48:41] DaBPunkt: s1-rr-a-wd: 21m 26s [+0.38 s/s]; s1-user-wd: 7m 55s [-2.05 s/s]; s2-user: 10s [-0.00 s/s]; s2-user-wd: 22m 19s [+0.37 s/s]; s3-rr-a: 21s [+0.00 s/s]; s3-user: 21s [+0.00 s/s]; s4-user-wd: 21m 53s [+0.27 s/s]; s5-user-wd: 22m 19s [+0.37 s/s] [16:51:31] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:56:37] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 56236 MB (5% inode=97%): [16:58:57] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (312640 kB) free! [17:00:57] Free Memory on turnera is WARNING: WARNING - 5.4% (451852 kB) free! [17:01:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (419544 kB) free! [17:06:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:06:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:07:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:08:09] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [17:11:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:11:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:14:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:15:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:17:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:21:59] Free Memory on turnera is WARNING: WARNING - 5.8% (484844 kB) free! [17:25:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:27:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:27:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:32:57] Free Memory on turnera is OK: OK - 7.3% (611108 kB) free. [17:42:58] Load avg. on adenia is WARNING: WARNING - load average: 13.95, 15.61, 13.63 [17:43:57] Free Memory on turnera is WARNING: WARNING - 5.3% (446168 kB) free! [17:43:58] Load avg. on adenia is OK: OK - load average: 12.41, 14.85, 13.49 [17:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:59:00] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (300288 kB) free! [18:01:01] Free Memory on turnera is WARNING: WARNING - 5.4% (452776 kB) free! [18:07:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:07:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:07:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:08:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [18:11:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:12:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:15:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:15:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:17:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:25:49] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:27:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:27:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:45:30] DaBPunkt: I would like to try and update an apache module (mod_tile) on ptolemy. Would you be the person to talk to, or would it better be nosy? [18:51:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:58:57] Free Memory on turnera is CRITICAL: CRITICAL - 3.4% (288724 kB) free! [19:03:59] Free Memory on turnera is WARNING: WARNING - 5.1% (427880 kB) free! [19:07:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:07:18] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:07:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:09:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [19:11:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:12:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:15:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:16:09] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:17:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:26:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:28:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:28:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:57:08] DaB. * [Toolserver-announce] Welcome on the Toolserver: Wikivoyage(s) [19:58:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.3% (279304 kB) free! [20:00:59] Free Memory on turnera is WARNING: WARNING - 5.1% (431508 kB) free! [20:02:00] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (404700 kB) free! [20:06:01] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196578 MB (20% inode=99%): [20:07:16] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:07:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:07:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:09:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [20:11:20] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:13:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:15:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:16:02] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:17:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:21:58] Free Memory on turnera is WARNING: WARNING - 5.7% (478508 kB) free! [20:23:28] @replag [20:23:28] DaBPunkt: s1-rr-a-wd: 2m 10s [-0.09 s/s]; s1-user-wd: 1m 59s [-0.03 s/s]; s2-user-wd: 1m 59s [-0.09 s/s]; s3-rr-a: 39s [+0.00 s/s]; s3-user: 39s [+0.00 s/s]; s4-user-wd: 2m 0s [-0.09 s/s]; s5-user-wd: 2m 0s [-0.09 s/s] [20:26:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:28:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:28:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:28:42] nighty o/ =___= [20:31:57] Free Memory on turnera is OK: OK - 7.1% (593188 kB) free. [20:39:15] Free Memory on turnera is WARNING: WARNING - 6.2% (522208 kB) free! [20:49:58] Free Memory on turnera is OK: OK - 7.1% (598280 kB) free. [21:07:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:07:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:07:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:09:11] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [21:11:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:13:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:13:42] @replag [21:13:44] DaBPunkt: s1-rr-a-wd: 18s [-0.04 s/s]; s1-user-wd: 18s [-0.03 s/s]; s2-user-wd: 20s [-0.03 s/s]; s3-rr-a: 1m 31s [+0.02 s/s]; s3-user: 1m 31s [+0.02 s/s]; s4-user-wd: 20s [-0.03 s/s]; s5-user-wd: 20s [-0.03 s/s] [21:14:31] DaBPunkt: Did you see my question from earlier regarding the apache module? [21:14:41] no [21:15:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:16:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:17:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:17:29] DaBPunkt: I would like to update an apache module (mod_tile) on ptolemy, and was wondering if I should rather talk to you or nosy about that? [21:18:34] apmon_: who updated it last time? [21:18:58] Not sure, I suspect that was still River [21:19:10] as it was last updated in 2010 if I am not mistaken [21:20:31] I feared that that would be teh answer. Let me look if I can find something [21:21:58] Free Memory on turnera is WARNING: WARNING - 5.9% (497936 kB) free! [21:26:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:28:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:28:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:28:23] Thanks. [21:31:58] Free Memory on turnera is OK: OK - 7.3% (610652 kB) free. [21:32:40] apmon_: I can't find anything in puppet or in our software-specs. Do you have a source? [21:33:26] source for what? Source code of the apache module? [21:33:41] yes [21:34:38] The original is at http://svn.openstreetmap.org/applications/utils/mod_tile/, but I have a couple of modifications in the source for compatibility of solaris [21:34:59] Those should be in ~osm/tools2/src/mod_tile [21:36:25] apmon_: do you think you are able to compile the mod yourself? [21:40:00] Free Memory on turnera is WARNING: WARNING - 5.6% (468856 kB) free! [21:45:56] DaBPunkt: I think I can. But I still have the issue with the stack protection [21:46:40] With the standard settings it doesn't compile complaining about a missing symbol __stack_chk_fail_local [21:47:24] I can turn off stack protection manually, but I don't know if that is an issue if apache is compiled in a different way than the module and I don't know of a way to test this other than installing the module [21:47:54] then lets test it [21:48:58] I'll need to see if I still have to correct setting to disable the stack protection handy. Otherwise I will probably need to do that tonight when I have a bit more time [21:49:39] I will be arround tomorrow after 16 CET again [21:49:58] It is also, given that mod_tile is used in production in a number of wikipedias, it will be important to make sure we have a backup of the old module, to be able to role back to in case it doesn't work [21:50:16] I already made a backup [21:50:37] OK, great, I'll try and compile it and check things as far as possible tonight then. [21:51:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:51:52] apmon_, it should be -fno-stack-protector [21:52:35] Platonides: If I remember correctly it was a little more complicated, as the option needed to be passed through the apxs apache build system [21:52:50] ouch [21:52:54] it may be, then [21:53:14] maybe setting it in $CFLAGS it gets passed? [21:55:50] Ah, it might be just -Wc,-fno-stack-protector [21:58:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (324012 kB) free! [22:00:57] Free Memory on turnera is WARNING: WARNING - 5.8% (485504 kB) free! [22:07:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:07:22] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:07:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:09:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [22:12:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:13:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:16:01] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:16:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:18:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:27:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:28:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:28:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:30:28] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 27441 MB (6% inode=99%): [22:30:29] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 55707 MB (5% inode=97%): [22:51:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:53:46] @replag [22:53:47] DaBPunkt: s1-rr-a-wd: 33m 41s [+0.33 s/s]; s1-user-wd: 33m 41s [+0.33 s/s]; s2-user-wd: 33m 41s [+0.33 s/s]; s3-rr-a: 24m 53s [+0.23 s/s]; s3-user: 24m 53s [+0.23 s/s]; s4-user-wd: 33m 41s [+0.33 s/s]; s5-user-wd: 33m 41s [+0.33 s/s] [22:54:19] @replag [22:54:24] DaBPunkt: s1-rr-a-wd: 33m 53s [+0.37 s/s]; s1-user-wd: 33m 53s [+0.37 s/s]; s2-user-wd: 33m 53s [+0.37 s/s]; s3-rr-a: 22m 48s [-3.83 s/s]; s3-user: 22m 48s [-3.83 s/s]; s4-user-wd: 33m 53s [+0.37 s/s]; s5-user-wd: 33m 53s [+0.37 s/s] [22:57:04] @replag [22:57:04] DaBPunkt: s1-rr-a-wd: 33m 59s [+0.04 s/s]; s1-user-wd: 34m 1s [+0.05 s/s]; s2-user-wd: 33m 59s [+0.04 s/s]; s3-rr-a: 8m 46s [-5.23 s/s]; s3-user: 8m 46s [-5.23 s/s]; s4-user-wd: 33m 59s [+0.04 s/s]; s5-user-wd: 33m 59s [+0.04 s/s] [22:58:09] @replag [22:58:10] DaBPunkt: s1-rr-a-wd: 33m 16s [-0.62 s/s]; s1-user-wd: 33m 20s [-0.59 s/s]; s2-user-wd: 33m 16s [-0.62 s/s]; s4-user-wd: 33m 17s [-0.60 s/s]; s5-user-wd: 33m 17s [-0.60 s/s] [22:58:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.8% (322348 kB) free! [23:06:25] @replag [23:06:25] DaBPunkt: s1-rr-a-wd: 21m 25s [-1.44 s/s]; s1-user-wd: 21m 58s [-1.38 s/s]; s2-user-wd: 21m 7s [-1.47 s/s]; s3-rr-a: 28s [-0.88 s/s]; s3-user: 28s [-0.88 s/s]; s4-user-wd: 21m 54s [-1.38 s/s]; s5-user-wd: 21m 7s [-1.48 s/s] [23:08:16] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:08:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:08:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:08:58] Free Memory on turnera is WARNING: WARNING - 5.2% (432688 kB) free! [23:09:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [23:09:15] @replag [23:09:16] DaBPunkt: s1-rr-a-wd: 17m 54s [-1.23 s/s]; s1-user-wd: 18m 49s [-1.11 s/s]; s2-user-wd: 17m 49s [-1.16 s/s]; s3-rr-a: 57s [+0.17 s/s]; s3-user: 57s [+0.17 s/s]; s4-user-wd: 18m 37s [-1.15 s/s]; s5-user-wd: 17m 49s [-1.16 s/s] [23:10:20] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (411036 kB) free! [23:12:19] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:12:50] @replag [23:12:50] DaBPunkt: s1-rr-a-wd: 1m 28s [-4.60 s/s]; s1-user-wd: 1m 28s [-4.86 s/s]; s2-user-wd: 1m 28s [-4.58 s/s]; s3-rr-a: 22s [-0.16 s/s]; s3-user: 22s [-0.16 s/s]; s4-user-wd: 1m 28s [-4.80 s/s]; s5-user-wd: 1m 28s [-4.58 s/s] [23:14:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:15:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:16:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:16:52] @replag [23:16:52] DaBPunkt: s1-rr-a-wd: 23s [-0.27 s/s]; s1-user-wd: 30s [-0.24 s/s]; s2-user-wd: 30s [-0.24 s/s]; s3-rr-a: 28s [+0.02 s/s]; s3-user: 28s [+0.02 s/s]; s4-user-wd: 23s [-0.27 s/s]; s5-user-wd: 30s [-0.24 s/s] [23:17:13] better [23:18:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:27:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:29:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:29:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:31:20] @replag [23:31:21] DaBPunkt: s1-rr-a-wd: 1m 33s [+0.08 s/s]; s1-user: 2m 21s [+0.00 s/s]; s1-user-wd: 2m 14s [+0.12 s/s]; s2-user-wd: 2m 26s [+0.13 s/s]; s3-rr-a: 2m 4s [+0.11 s/s]; s3-user: 2m 4s [+0.11 s/s]; s4-user-wd: 3m 41s [+0.23 s/s]; s5-user-wd: 2m 26s [+0.13 s/s] [23:32:02] @replag [23:32:02] DaBPunkt: s1-rr-a-wd: 1m 26s [-0.17 s/s]; s1-user: 2m 47s [+0.63 s/s]; s1-user-wd: 1m 26s [-1.16 s/s]; s2-user-wd: 1m 26s [-1.45 s/s]; s3-rr-a: 39s [-2.06 s/s]; s3-user: 39s [-2.06 s/s]; s4-user-wd: 1m 26s [-3.27 s/s]; s5-user-wd: 1m 26s [-1.46 s/s] [23:33:30] @replag [23:33:30] DaBPunkt: s1-rr-a-wd: 1m 10s [-0.18 s/s]; s1-user: 3m 45s [+0.66 s/s]; s1-user-wd: 1m 10s [-0.18 s/s]; s2-user-wd: 1m 10s [-0.18 s/s]; s3-rr-a: 21s [-0.20 s/s]; s3-user: 21s [-0.20 s/s]; s4-user-wd: 1m 10s [-0.18 s/s]; s5-user-wd: 1m 10s [-0.18 s/s] [23:44:38] @replag [23:44:39] DaBPunkt: s1-rr-a-wd: 1m 51s [+0.06 s/s]; s1-user: 5m 46s [+0.18 s/s]; s1-user-wd: 1m 52s [+0.06 s/s]; s2-user-wd: 1m 51s [+0.06 s/s]; s3-rr-a: 3m 40s [+0.30 s/s]; s3-user: 3m 40s [+0.30 s/s]; s4-user-wd: 1m 51s [+0.06 s/s]; s5-user-wd: 1m 51s [+0.06 s/s] [23:49:46] @replag [23:49:47] DaBPunkt: s1-rr-a-wd: 60s [-0.17 s/s]; s1-user: 50s [-0.96 s/s]; s1-user-wd: 60s [-0.17 s/s]; s2-user-wd: 60s [-0.17 s/s]; s3-rr-a: 2m 1s [-0.32 s/s]; s3-user: 2m 1s [-0.32 s/s]; s4-user-wd: 60s [-0.17 s/s]; s5-user-wd: 60s [-0.17 s/s] [23:51:27] @replag [23:51:27] DaBPunkt: s1-rr-a-wd: 2m 36s [+1.00 s/s]; s1-user-wd: 2m 36s [+1.00 s/s]; s2-user: 25s [+0.00 s/s]; s2-user-c: 28s [+0.00 s/s]; s2-user-wd: 2m 37s [+1.00 s/s]; s3-rr-a: 3m 38s [+1.00 s/s]; s3-user: 3m 38s [+1.00 s/s]; s4-rr-a: 11s [+0.00 s/s] [23:51:27] DaBPunkt: s4-user: 11s [+0.00 s/s]; s4-user-wd: 2m 37s [+1.00 s/s]; s5-user-c: 30s [+0.00 s/s]; s5-user-wd: 2m 38s [+1.00 s/s] [23:51:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:57:36] @replag [23:57:36] DaBPunkt: s1-rr-a-wd: 1m 54s [-0.11 s/s]; s1-user-c: 12s [+0.00 s/s]; s1-user-wd: 1m 54s [-0.11 s/s]; s2-user-c: 12s [-0.04 s/s]; s2-user-wd: 1m 54s [-0.12 s/s]; s3-rr-a: 41s [-0.47 s/s]; s3-user: 41s [-0.47 s/s]; s4-rr-a: 12s [+0.00 s/s] [23:57:37] DaBPunkt: s4-user: 12s [+0.00 s/s]; s4-user-wd: 1m 54s [-0.12 s/s]; s5-user-c: 12s [-0.05 s/s]; s5-user-wd: 1m 54s [-0.12 s/s]