[00:00:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82305 MB (13% inode=99%): [00:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:55:37] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1957 [00:56:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1966.000000 [01:00:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:07:06] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [01:07:37] MySQL slave on rosemary is OK: Uptime: 15241807 Threads: 12 Questions: 6762573109 Slow queries: 2167093 Opens: 306330 Flush tables: 6 Open tables: 4149 Queries per second avg: 443.685 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1755 [01:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82153 MB (13% inode=99%): [01:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:00:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82028 MB (13% inode=99%): [02:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:12:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:38:56] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:54:26] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38387 MB (9% inode=99%): [02:54:37] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71141 MB (7% inode=98%): [02:54:37] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71141 MB (7% inode=98%): [03:00:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:08:27] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81907 MB (13% inode=99%): [03:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:22:08] / on damiana is WARNING: DISK WARNING - free space: / 14126 MB (19% inode=95%): [03:42:41] how does one launch several screens via a bash file in parallel? [03:43:07] / on damiana is OK: DISK OK - free space: / 24139 MB (33% inode=95%): [03:45:15] What problem are you trying to solve? [03:46:42] I want to launch several bots a once which are all normally kept in separate screens [03:47:52] IE bash file: [03:47:56] screen bot1 [03:47:59] screen bot2 [03:51:31] Brooke: any ideas? [03:53:17] I'm not sure why you'd want to run bots in screen. [03:53:39] I use screen when I need to wget a large file. That's pretty much it. [03:53:50] Or run a dump script, I guess. [04:00:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81760 MB (13% inode=99%): [04:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:00:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81594 MB (13% inode=99%): [05:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:45:54] @replag [05:46:04] liangent: s2-user: 1d 1h 34m 59s [-0.67 s/s]; s3-rr-a: 5m 28s [+0.01 s/s]; s3-user: 5m 28s [+0.01 s/s] [06:00:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81458 MB (13% inode=99%): [06:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:12:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:31:17] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:00:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81305 MB (13% inode=99%): [07:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:31:17] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:00:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81103 MB (13% inode=99%): [08:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:31:16] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:00:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:08:05] Dr. Trigon * Re: [Toolserver-l] Future of the toolserver [09:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80981 MB (13% inode=99%): [09:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:12:56] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:12:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:13:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:16:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 104148 MB (10% inode=99%): [09:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:31:17] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:50:04] Dr. Trigon * Re: [Toolserver-l] Switching of SGE-arch-default at 9. October [10:00:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80821 MB (13% inode=99%): [10:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:12:57] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:31:16] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:00:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:12:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80668 MB (13% inode=99%): [11:12:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:12:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:12:56] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:12:57] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:12:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:13:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:13:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:13:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:13:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:17:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:31:17] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:43:56] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:44:06] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:44:06] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38106 MB (9% inode=99%): [11:46:37] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 71677 MB (7% inode=98%): [11:46:37] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 71677 MB (7% inode=98%): [12:00:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:12:37] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80551 MB (13% inode=99%): [12:12:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:12:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:13:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:13:07] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:13:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:13:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:13:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:13:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:13:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:13:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:17:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:19:16] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:07:50] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2101.000000 [13:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80319 MB (13% inode=99%): [13:23:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:24:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:07:50] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3387.000000 [14:14:30] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2172 [14:15:01] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2209.000000 [14:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80117 MB (13% inode=99%): [14:23:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:24:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:41:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [14:41:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3631 [14:50:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [15:00:06] Maarten Dammers * [Toolserver-l] Wikimedia Nederland Hackathon 2012 [15:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:10:47] Ouch, raid battery about to die? :-( [15:12:10] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2211.000000 [15:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79949 MB (13% inode=99%): [15:23:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:24:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:31:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3586.000000 [15:41:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6136.000000 [15:41:30] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6158 [16:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:10:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [16:10:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [16:14:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79756 MB (13% inode=99%): [16:23:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:24:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:41:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8443.000000 [16:41:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8465 [16:49:20] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:10:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6962.000000 [17:10:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6806.000000 [17:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79557 MB (13% inode=99%): [17:23:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:24:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:29:05] Dr. Trigon * Re: [Toolserver-l] Future of the toolserver [17:41:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10806.000000 [17:41:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10826 [17:50:05] Dr. Trigon * Re: [Toolserver-l] Future of the toolserver [18:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:07:27] Merlissimo, are you here? [18:07:38] Platonides: yes [18:08:28] abnout the email, I was indeed thinking in modifying the epilog script [18:09:41] I think it would be something like if qstat -j $JOB_ID | grep -q [18:09:41] mail_options:.*e [18:10:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8467.000000 [18:10:32] in the epilog script you can request nearly all option directly [18:10:49] how so? [18:10:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9844.000000 [18:11:10] I did look if they were in some env var, but didn't found it [18:11:12] the question is only how to activate this option by setting a resource value or envorionment oder something else [18:11:49] my idea was to send the output by email when the user requested to be notified of the job finish by email [18:15:02] assuming that it isn't a feature too useful, and thus we can replace it with that new meaning without problems [18:15:08] the question is also what about job that use a log shared by multiple tasks or jobs [18:15:11] you cna probably check if people have it set [18:15:45] what I had thought was simply in the epilog [18:15:50] after it removed the empty files [18:15:53] if you had that option [18:16:04] send the content of that file by email and remove the file [18:16:07] as you maybe have read from config i already remove empty output files [18:16:48] yes, I read the epilog file [18:17:05] although I don't remember now where was that file [18:19:08] everything special is in /sge/scripts [18:19:51] ah, yes [18:19:58] /sge/scripts/epilog.sh [18:20:41] ah, epilog-lx.sh is now different [18:20:50] it used to be the same as the previous one [18:21:38] btw, what's the point of that sleep 5 ? [18:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79153 MB (12% inode=99%): [18:23:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:24:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:16] that because some jobs are very short and sometimes the scheduler did not regocnize that a job has startet because reading pid from nfs took longer than the job runtime because of nfs problems [18:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:41:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13159.000000 [18:41:30] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13183 [19:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:10:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8435.000000 [19:10:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12396.000000 [19:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78890 MB (12% inode=99%): [19:23:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:41:00] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [19:41:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14825 [20:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:10:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8212.000000 [20:10:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15833.000000 [20:23:31] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78525 MB (12% inode=99%): [20:23:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:41:30] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16466 [20:41:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16479.000000 [20:56:10] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3363.000000 [21:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:02:11] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1608.000000 [21:10:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19333.000000 [21:17:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 104045 MB (10% inode=99%): [21:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78295 MB (12% inode=99%): [21:23:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:23:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:28:28] Is SGE not packaged as .deb on the Debian servers? [21:41:30] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13447 [21:41:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13415.000000 [22:01:01] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:09:10] scfc_de: no, why do you need it? [22:10:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22647.000000 [22:23:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78138 MB (12% inode=99%): [22:23:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:23:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:23:50] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:23:50] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:24:10] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:24:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:24:20] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:24:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:41:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8303 [22:41:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8304.000000 [22:57:11] Merlissimo: I wonder how it is maintained. I would assume anything but .debs would be a major threat to mental health :-). [22:58:18] i compiled solaris and linux2.6 binaries from source on solaris boxes [22:59:25] we need the same version on solaris and linux, so we cannot use any precompiled debs for debian only [23:00:42] Well, it is running :-). I personally always opt for packages, as the build process is easily reproducible and patches are clearly distinguishable from the original source. [23:01:00] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:01:33] Is the /sge directory shared between the hosts? [23:02:09] yes [23:03:47] its the same for linux and solaris and it also contains the master spool because it is needed for failover [23:07:27] because i build sge on solaris i think linux does not contain all header files needed for compiling (a added many on solaris) [23:08:18] Header files? I don't see any beneath /sge (except /sge/GE/include/drmaa.h). [23:09:32] You are a Solaris admin, aren't you? Do you have a pointer to an introduction in its packaging system for someone used to Linux? [23:10:00] Not necessarily enough to build packages, but to understand Solaris' concept with that regard. [23:10:32] i compled sge at /mnt/user-store/sge [23:10:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25669.000000 [23:11:19] i simply compiled the binaries and installed them without any packaging [23:12:12] But do you know how packaging in Solaris works? [23:12:23] yes [23:12:32] but i am not a solaris admins [23:13:26] Do you know some introduction, then? [23:14:20] i read a book maybe 10 years ago [23:14:36] ;-) [23:14:53] Okay, that's a bit too much effort :-). I'll see what Google has to offer :-). [23:15:20] just search for pkg, pkginfo and so on [23:18:04] the main workflow is to use prototype as input pkgmk [23:20:59] That's a lot of stuff to read :-). [23:23:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78011 MB (12% inode=99%): [23:23:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:23:41] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:23:51] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:23:51] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:24:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:24:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:24:21] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:24:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:24:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:41:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6289 [23:41:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6269.000000 [23:47:20] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37304 MB (9% inode=99%): [23:47:30] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 72768 MB (7% inode=98%): [23:47:30] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 72768 MB (7% inode=98%):