[00:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65826 MB (10% inode=99%): [00:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:29:49] @replag [00:29:50] scfc_de: s1-rr-a: 17m 42s [+0.59 s/s]; s1-user: 17m 42s [+0.59 s/s]; s3-rr-a: 1m 48s [+0.03 s/s]; s3-user: 1m 48s [+0.03 s/s] [00:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:00:33] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2041 [01:01:04] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2053.000000 [01:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65706 MB (10% inode=99%): [01:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:00:33] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2020 [02:01:04] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2027.000000 [02:11:03] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1775.000000 [02:11:34] MySQL slave on rosemary is OK: Uptime: 15591248 Threads: 11 Questions: 6919158037 Slow queries: 2285941 Opens: 334137 Flush tables: 6 Open tables: 4159 Queries per second avg: 443.784 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1710 [02:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65598 MB (10% inode=99%): [02:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:25:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:44:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:54:53] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65491 MB (10% inode=99%): [03:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:26:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65381 MB (10% inode=99%): [04:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:26:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:22:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65266 MB (10% inode=99%): [05:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:26:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65126 MB (10% inode=99%): [06:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:26:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64963 MB (10% inode=99%): [07:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64730 MB (10% inode=99%): [08:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:17:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 103186 MB (10% inode=99%): [09:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64496 MB (10% inode=99%): [09:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:08:03] Damianz, ping [10:08:09] oops [10:08:14] Danny_B|backup, ping [10:08:32] just woke up [10:14:42] oh ok :D me too [10:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64220 MB (10% inode=99%): [10:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:29:33] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196290 MB (20% inode=99%): [10:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:56:23] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35559 MB (8% inode=99%): [10:56:34] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71747 MB (7% inode=98%): [10:56:34] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71747 MB (7% inode=98%): [11:28:26] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:26] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:28:38] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:40] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71698 MB (7% inode=98%): [11:28:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71698 MB (7% inode=98%): [11:28:58] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36294 MB (8% inode=99%): [11:28:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:29:23] Free Memory on damiana is CRITICAL: CRITICAL - 3.4% (286356 kB) free! [11:30:24] Free Memory on damiana is WARNING: WARNING - 7.0% (589132 kB) free! [11:37:19] Free Memory on damiana is OK: OK - 8.2% (689004 kB) free. [11:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:00:24] Free Memory on damiana is WARNING: WARNING - 5.3% (443512 kB) free! [12:02:24] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (399940 kB) free! [12:04:23] Free Memory on damiana is WARNING: WARNING - 5.2% (434504 kB) free! [12:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63675 MB (10% inode=99%): [12:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:27:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:42:33] Danny_B|backup, awake? [13:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63521 MB (10% inode=99%): [13:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:23:56] HardDisk_WP: is TS-1540 from you? [13:24:15] i filed a ticket, but dunno which ID [13:24:17] hold short... [13:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:25:01] DaBPunkt, indeed it is [13:25:10] but i found the private-key agai [13:25:18] so i only need account reactivated [13:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:28:33] HardDisk_WP: ok, wait a minute [13:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:55:36] DaBPunkt, thanks :) [13:55:50] wow shit, it's been two years since i worked on the static dumps last time... [13:57:02] you should re-read the rules and maybe take a look at the announce-list-archive [14:01:23] nothing new in the rules, but i'll take a look at the announce list. let me guess, the entire DB-server infrastructure has changed? [14:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63214 MB (10% inode=99%): [14:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:25:24] Free Memory on damiana is WARNING: WARNING - 6.9% (580356 kB) free! [14:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:28:34] DaBPunkt, uh... i try to access the database dewiki_p on dewiki.rrdb.toolserver.org, but i get database not found error [14:28:48] use dewiki_p; [14:28:48] ERROR 1049 (42000): Unknown database 'dewiki_p' [14:29:54] HardDisk_WP: dewiki-p.rrdb.toolserver.org [14:30:15] ah k, thanks [14:31:23] Free Memory on damiana is OK: OK - 7.7% (641588 kB) free. [14:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:53:54] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:54:04] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:54:04] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:55:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36036 MB (8% inode=99%): [14:55:34] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73217 MB (7% inode=98%): [14:55:34] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73217 MB (7% inode=98%): [15:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62883 MB (10% inode=99%): [15:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:12:00] DaBPunkt, uh, can you please update git on willow? [16:12:11] apparently it doesn't support git push to github https [16:12:50] see http://pastebin.com/1brwhVYk for details [16:13:21] also, the manpages aren't installed, neither man git-push nor git help push works [16:14:29] HardDisk_WP: Have you tried it on the Linux boxes? [16:15:33] nope...hold short [16:16:28] nightshade works [16:17:44] You should file a ticket for willow, though. (Though I hope that it is switched to Linux soon :-).) [16:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62599 MB (10% inode=99%): [16:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62371 MB (10% inode=99%): [17:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:30:34] Free Memory on damiana is WARNING: WARNING - 5.1% (426800 kB) free! [17:31:24] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (367832 kB) free! [17:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:32:24] Free Memory on damiana is WARNING: WARNING - 7.0% (583160 kB) free! [17:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:56:26] DaBPunkt: TS-1542 falls du mal reinschauen magst [18:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62104 MB (10% inode=99%): [18:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:04:58] hello [19:05:19] i will do the maintenance of the sql-user-databases today [19:05:32] this means they will go offline in some minutes [19:08:24] ok i will start now [19:15:01] nosy: you forgot to update status. [19:15:04] MySQL on adenia is CRITICAL: Cant connect to MySQL server on adenia (146) [19:15:34] giftpflanze: "[10/Oct/2012:19:12:41 +0000] WARN:toolserver.org:CGI: Request to execute script '/home/gifti/public_html/tool/toterlink.cgi' denied. HTaccess files have restricted the use of CGIs in this directory." [19:16:58] dabpunkt: oh thanks [19:17:14] you already set the status? [19:17:41] yes [19:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61793 MB (10% inode=99%): [19:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:38:36] i've a problem with password restoring... any admin who can help me? [19:39:33] DaBPunkt: where is that from?[2~ [19:41:11] well, this is cgi, not fcgi, so it is unrelated [19:41:25] but i wonder where it comes from [19:41:48] giftpflanze: /var/log/http/errors [19:42:04] Gnumarcoo: what is the problem? [19:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:44:00] DaBPunkt: password recovering doesn't work: i'f i login with temporary psw, my username... is my IP. So when I confirm a new password, an error occourred: "no such user". Can you restore it for me? [19:44:41] Gnumarcoo: jira, wiki or login-server? [19:45:49] wiki and login-server... [19:45:53] DaBPunkt, ^ [19:46:26] Gnumarcoo: that's 2 differend systems [19:46:46] I know [19:47:20] you can not reset the password in one system and use that on the other [19:47:51] nah... restore only wiki, please [19:48:22] I can't. The system is broken at the momemt [19:48:30] nosy: What is the crows-status? [19:48:33] anything new? [19:48:53] DaBPunkt: what is crows? [19:48:55] DaBPunkt, when is possible, can you do it? [19:49:04] seriously does not ring any bell here [19:49:11] nosy: I mean the problem with JIRA [19:49:23] ah crowd...i remember... [19:49:30] i had no time to look at it [19:49:50] Gnumarcoo: if it is working you can do it yourself :) [19:49:53] but probably i have some now [19:50:12] * Damianz moos [19:50:23] DaBPunkt, isn't the first time that psw recover doesn't work. [19:50:36] mmh I'll try it. [20:05:23] Free Memory on damiana is WARNING: WARNING - 5.2% (432252 kB) free! [20:11:24] Free Memory on damiana is OK: OK - 7.2% (601304 kB) free. [20:15:04] MySQL on adenia is CRITICAL: Cant connect to MySQL server on adenia (146) [20:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61568 MB (10% inode=99%): [20:23:03] MySQL on adenia is OK: Uptime: 69 Threads: 12 Questions: 2863 Slow queries: 13 Opens: 120 Flush tables: 1 Open tables: 113 Queries per second avg: 41.492 [20:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:24:02] maintenance finished [20:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:27:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:17:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 100172 MB (10% inode=99%): [21:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61305 MB (10% inode=99%): [21:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:28:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:28:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:35:10] nosy, I think you should at least have sent an email to toolserver-l about it [21:35:27] I don't understand why this wasn't announced before [21:40:42] ping Merlissimo [21:40:53] ? [21:41:08] Platonides: what was not announced? [21:41:38] DaBPunkt, shutdown of sql cluster? [21:41:41] or did I miss it? [21:41:45] was announced [21:42:01] [Toolserver-l] Maintenance sql-user / host adenia 10th October [21:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:42:15] ah, right [21:42:24] sorry [21:42:31] np [21:42:59] Merlissimo, the time given by qstat is in UTC? [21:43:01] nosy: how far are you? your window ended 42 minutes ago [21:43:15] Platonides: yes, of course [21:43:37] well, I'm wondering why I job is still waiting [21:43:45] it was programmed to start on "10/10/2012 21:28:01" [21:44:00] local run of date says it's " 21:43:46" [21:44:13] maybe there is no memory :/ [21:44:28] execution_time: Wed Oct 10 22:25:00 2012 [21:44:29] I precisely want to see how much memory it uses [21:44:50] Merlissimo? [21:45:00] "execution time not reached" [21:45:24] Merlissimo, where are you viewing that? [21:45:25] Platonides: that what you job "rm" says [21:45:34] qstat -j 548958 [21:46:00] this is what I'm viewing: http://pastebin.com/Wcf8phX9 [21:46:41] and for job detail you can request "qstat -j " [21:46:42] Platonides: the first number is the job-id. You can seee details with qstat -j *jobid* [21:46:55] * DaBPunkt let Merlissimo do the explanation :) [21:47:04] I guess that's the submission time, not when it started... [21:47:09] that header is misleading [21:47:32] the header has a line break [21:47:46] Platonides: it's 21:46 UTC, so 22:55 can not be the submission-time [21:48:12] DaBPunkt, I refer to the output of plain qstat [21:48:15] for colums is job id and last column is ja-task-id [21:48:32] you have to buy a new wider display ;-) [21:52:23] Platonides: have you seen you job details? [21:53:47] yes [21:53:50] thanks [21:54:35] your job run time is less than one second. is this expected? [21:54:35] I never remember qstat parameters [21:54:47] it's just a rm -r, that's expected :) [21:56:14] then you should also the maximum run time, because default is 6 hours [21:56:27] it is set now [21:57:05] DaBPunkt, I see an strange error message when connecting to mayapple: [21:57:06] key_read: uudecode AAAAB3NzaC1yc2EAAAADAQABAAABAQDqnHyI3/cgzqZn2LZq57VjtMGpqDzzjV/+OCFfdxbxCKMlpEktvyrTXNCQ2FlQzHYJudZkdsmSilwBIc3EDNZVKN5o8+5T4yHo+Hfb8YaAeeq0cq21kxuxBgVJRNdNPsNfBG0ElbO03WKv9z9dFCoL+fKryUgKKAJ6um4MBydo56awWtP3ynxcRcvtKN/nII7NMLbprBYYPMqgYQKJSMYIBl5ODdp84Velod9FvmgX0gDI3mCpISQkIrNRY1HsheJQwXZSU+0CxsDIHp7pBoJHtaBFLL7bW5oJ0nPmExrQgwyNS+rJ6p9bZh95D040vfGcGcwuT05YScb1OIYNj6mjwolfsbane.toolserver.org ecdsa-sha2-nistp256 [21:57:40] wait, that was in my local known_hosts [21:57:48] * Platonides slaps Platonides around a bit with a large trout [21:58:08] btw, what's the story of that host? [22:06:42] it is not possible to submti jobs from the web servers :( [22:06:49] "Unable to run job: denied: client (ortelius.toolserver.org/qsub/1) uses old GDI version 6.2u4 while qmaster uses newer version 2011.11" [22:19:01] Platonides: which command did you use? [22:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60953 MB (9% inode=99%): [22:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:29:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:30:30] Platonides: works for me [22:31:17] have you set you path manually? [22:32:09] env -i PATH="/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin" SGE_ROOT=/sge/GE SGE_QMASTER_PORT=536 qsub -a 201210110010.10 -wd /home/platonides/ -j [22:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:32:28] (that value of PATH is the one used by the web server) [22:33:14] * DaBPunkt is working at the moment to remove old /sge62… [22:42:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:43:26] hmm, running env -i SGE_CLUSTER_NAME=toolserver PATH="/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin" SGE_ROOT=/sge/GE SGE_QMASTER_PORT=536 /sge/GE [22:43:47] the problem is in that the web server runs with a $PATH pointing to the old sge install [22:44:25] and those paths like /sge/GE/bin/sol-amd64/ shouldn't be hardcoded, as they are architecture dependant :S [22:45:46] that path is wrong [22:46:24] you mean the PATH="/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin" ?? [22:46:43] it's the one that web server scripts get [22:51:14] opened as ticket https://jira.toolserver.org/browse/TS-1543 [22:53:49] Platonides: for TS-1505: do you know how? [22:57:28] DaBPunkt my idea was to use ulimit [22:58:34] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [22:58:51] the command ulimit -c 0 [22:59:00] in the shell that spawns the web server should do it [22:59:52] in /etc/init.d/zeus I guess? [22:59:54] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [23:02:43] Platonides: which shell do you use? [23:03:08] bash [23:03:51] although it seems to be the same command for eg. zsh [23:04:37] just for exmplanation: /sge62/bin/sol-amd64 is wrong it should be /sge/GE/bin/sol-amd64 . but on my shell@othelius it is correct [23:06:01] but for this you have to talk to DaBPunkt [23:06:05] Merlissimo: I assume you meant ortelius? [23:06:32] yes (it's too late) [23:07:22] Platonides: ok, try now [23:11:34] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [23:18:55] running ulimit -a suggests it won't create more core [23:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60691 MB (9% inode=99%): [23:23:17] Platonides: I was more interrested if qsub is working now [23:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:42:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:51:40] nacht ts [23:58:33] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [23:59:54] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat