[00:03:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:04:26] Free Memory on turnera is WARNING: WARNING - 5.1% (424992 kB) free! [00:05:26] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3508.000000 [00:07:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (372332 kB) free! [00:12:26] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1531.000000 [00:19:25] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107152 MB (11% inode=99%): [00:20:25] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106726 MB (10% inode=99%): [00:21:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107046 MB (11% inode=99%): [00:21:46] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2067.000000 [00:21:46] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2072 [00:24:55] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:29:26] Free Memory on turnera is WARNING: WARNING - 5.5% (463628 kB) free! [00:30:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:31:17] Is there an easy way with qsub to have standard output/error mailed instead of written to disk? Something like "at" on Linux? [00:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:40:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:40:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40730 MB (6% inode=99%): [00:46:25] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (308688 kB) free! [00:47:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:47:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:48:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 496637 MB (9% inode=43%): [00:48:25] Free Memory on turnera is WARNING: WARNING - 5.8% (487756 kB) free! [00:50:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:51:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:52:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:54:26] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:55:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:57:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105962 MB (10% inode=99%): [01:01:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108023 MB (11% inode=99%): [01:03:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:06:25] Free Memory on turnera is OK: OK - 7.3% (610116 kB) free. [01:06:47] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1777.000000 [01:06:47] MySQL slave on rosemary is OK: Uptime: 14809761 Threads: 15 Questions: 6574830657 Slow queries: 2023346 Opens: 278713 Flush tables: 6 Open tables: 3819 Queries per second avg: 443.952 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [01:17:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (369076 kB) free! [01:30:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:31:26] Free Memory on turnera is WARNING: WARNING - 5.1% (431520 kB) free! [01:32:26] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (415128 kB) free! [01:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:39:26] Free Memory on turnera is WARNING: WARNING - 5.3% (440184 kB) free! [01:40:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:40:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40585 MB (6% inode=99%): [01:44:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106869 MB (10% inode=99%): [01:47:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:48:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 496889 MB (9% inode=43%): [01:48:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:51:25] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108027 MB (11% inode=99%): [01:51:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:53:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:55:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:26] Free Memory on turnera is OK: OK - 7.1% (594884 kB) free. [02:03:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:04:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (300288 kB) free! [02:07:25] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106917 MB (10% inode=99%): [02:10:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107189 MB (11% inode=99%): [02:11:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106698 MB (10% inode=99%): [02:19:25] Free Memory on turnera is WARNING: WARNING - 5.1% (429228 kB) free! [02:31:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:34:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107451 MB (11% inode=99%): [02:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:40:25] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106808 MB (10% inode=99%): [02:40:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:41:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40408 MB (6% inode=99%): [02:44:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107700 MB (11% inode=99%): [02:46:26] Free Memory on turnera is OK: OK - 7.5% (630340 kB) free. [02:47:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:48:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 494881 MB (9% inode=42%): [02:49:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:51:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:53:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:55:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:03:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:09:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106910 MB (10% inode=99%): [03:32:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:39:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:40:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:42:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40231 MB (6% inode=99%): [03:43:48] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (307840 kB) free! [03:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:48:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 493909 MB (9% inode=42%): [03:49:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:51:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:51:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107063 MB (11% inode=99%): [03:51:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:52:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106485 MB (10% inode=99%): [03:54:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:55:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:04:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:11:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107381 MB (11% inode=99%): [04:12:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106432 MB (10% inode=99%): [04:12:47] Free Memory on turnera is WARNING: WARNING - 6.1% (510440 kB) free! [04:16:49] Free Memory on turnera is OK: OK - 7.9% (661648 kB) free. [04:17:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107539 MB (11% inode=99%): [04:32:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:38:48] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (300380 kB) free! [04:40:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:40:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:42:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40046 MB (6% inode=99%): [04:43:48] Free Memory on turnera is WARNING: WARNING - 5.6% (470672 kB) free! [04:47:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:49:47] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 495406 MB (9% inode=42%): [04:50:00] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:50:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:51:49] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (364812 kB) free! [04:52:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:54:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:55:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:57:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105205 MB (10% inode=99%): [05:04:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:14:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107447 MB (11% inode=99%): [05:32:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:41:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:41:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:42:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39806 MB (6% inode=99%): [05:47:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:49:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 495053 MB (9% inode=42%): [05:50:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:51:00] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:51:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:52:47] Free Memory on turnera is CRITICAL: CRITICAL - 2.7% (226580 kB) free! [05:53:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:55:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:04:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:05:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106859 MB (10% inode=99%): [06:06:18] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 7469 MB (0% inode=92%): [06:06:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107140 MB (11% inode=99%): [06:09:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106848 MB (10% inode=99%): [06:32:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:39:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107290 MB (11% inode=99%): [06:41:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:41:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:42:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39640 MB (6% inode=99%): [06:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:49:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 495208 MB (9% inode=42%): [06:50:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:51:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:52:48] Free Memory on turnera is CRITICAL: CRITICAL - 3.1% (257296 kB) free! [06:52:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:53:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:55:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:04:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:09:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106924 MB (10% inode=99%): [07:16:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107358 MB (11% inode=99%): [07:17:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106770 MB (10% inode=99%): [07:32:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:35:51] hi [07:37:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107253 MB (11% inode=99%): [07:41:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:41:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:42:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39505 MB (6% inode=99%): [07:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:50:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 495016 MB (9% inode=42%): [07:50:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:51:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:52:59] Free Memory on turnera is CRITICAL: CRITICAL - 3.8% (315792 kB) free! [07:53:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:54:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:55:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:04:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106792 MB (10% inode=99%): [08:07:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107234 MB (11% inode=99%): [08:33:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:33:59] Free Memory on turnera is WARNING: WARNING - 5.3% (445500 kB) free! [08:36:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (404536 kB) free! [08:37:59] Free Memory on turnera is WARNING: WARNING - 5.1% (426400 kB) free! [08:41:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:41:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:42:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39342 MB (6% inode=99%): [08:47:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:49:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 494789 MB (9% inode=42%): [08:51:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:51:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:53:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:53:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:55:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:05:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:10:37] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39606 MB (9% inode=99%): [09:10:59] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 78557 MB (8% inode=98%): [09:11:08] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 78572 MB (8% inode=98%): [09:15:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (380532 kB) free! [09:42:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39153 MB (6% inode=99%): [09:42:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:42:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:43:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106925 MB (10% inode=99%): [09:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:49:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 494258 MB (9% inode=42%): [09:52:00] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:52:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:54:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:54:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:55:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:05:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:16:00] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (236400 kB) free! [10:24:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106994 MB (11% inode=99%): [10:25:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106376 MB (10% inode=99%): [10:26:19] Is there an option to have qsub/qcronsub mail me the output of a command instead of writing it to disk? Something similar to regular cron/at? [10:34:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:42:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:42:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:43:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38953 MB (6% inode=99%): [10:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:50:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 494031 MB (9% inode=42%): [10:52:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:52:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:54:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:54:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:56:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:04:25] Tim Landscheidt * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [11:05:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:08:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107671 MB (11% inode=99%): [11:09:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106667 MB (10% inode=99%): [11:10:28] Tim Landscheidt * [Toolserver-l] How to have qsub mail output? [11:13:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107156 MB (11% inode=99%): [11:15:25] JamesR * Re: [Toolserver-l] How to have qsub mail output? [11:16:59] Free Memory on turnera is CRITICAL: CRITICAL - 3.1% (257320 kB) free! [11:22:25] Tim Landscheidt * Re: [Toolserver-l] How to have qsub mail output? [11:34:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:35:26] Platonides * Re: [Toolserver-l] How to have qsub mail output? [11:42:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:43:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:43:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107806 MB (11% inode=99%): [11:44:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38696 MB (6% inode=99%): [11:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:52:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 493463 MB (9% inode=42%): [11:52:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:53:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:54:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:55:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:56:26] Platonides * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [11:56:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:06:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:12:25] Tim Landscheidt * Re: [Toolserver-l] Reasons for not migrating to Tool Lab [12:14:25] Platonides * Re: [Toolserver-l] Future of the toolserver [12:16:59] Free Memory on turnera is CRITICAL: CRITICAL - 2.5% (207440 kB) free! [12:20:25] Tim Landscheidt * Re: [Toolserver-l] How to have qsub mail output? [12:34:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:40:08] hello all [12:40:25] Danny_B|backup: Did you see my eMail about reboots today? [12:42:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:44:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:45:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38465 MB (6% inode=99%): [12:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:52:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 493221 MB (9% inode=42%): [12:52:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:54:00] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:55:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:56:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:56:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:59:17] DaBPunkt: yup, why? [13:00:13] just to be sure that we kill not an open editor of you like the last time [13:03:04] ah, thanks [13:03:22] yeah, i'll have only bots running at that time [13:03:36] speaking about bots - does @reboot finally work or not yet? [13:03:40] DaBPunkt: ^^ [13:04:45] Danny_B|backup: I tested it successfully on the linux-boxes. But for normal you shoudl use SGE for bots and start them from a submit-host [13:06:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:09:08] DaBPunkt: how is it going with reimport of some corrupted databases i wrote in jira? [13:09:32] Danny_B|backup: The wmf is ignoring my email about dumps [13:09:59] who is? [13:11:42] I tried binasher and the general ops-requests@-mailadress [13:12:54] that's pity :-/ [13:14:51] I have to leave for 15 minutes. cu [13:16:59] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (177308 kB) free! [13:34:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:35:14] re [13:42:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:44:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:45:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38187 MB (6% inode=99%): [13:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:53:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 492917 MB (9% inode=42%): [13:53:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:54:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:55:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:56:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:56:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:07:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:17:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (379976 kB) free! [14:35:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:42:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:44:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:45:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37941 MB (6% inode=99%): [14:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:49:58] Hi, how i can renew my LPDA password? [14:49:59] Free Memory on turnera is WARNING: WARNING - 5.7% (475288 kB) free! [14:50:20] DaBPunkt: ^^ ? [14:50:31] renew? [14:51:15] i forgot my pass [14:51:26] the ldap-password does not expire. Do you forgot it? [14:51:39] yes i am [14:51:51] coet: no problem. run resetpass on willow [14:51:52] *i do [14:51:59] ok [14:52:04] thx [14:52:07] np [14:52:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 492713 MB (9% inode=42%): [14:53:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:55:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:56:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:56:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:57:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:59:00] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (409512 kB) free! [15:00:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106625 MB (10% inode=99%): [15:00:59] Free Memory on turnera is WARNING: WARNING - 5.1% (431512 kB) free! [15:03:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108404 MB (11% inode=99%): [15:08:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:20:59] Free Memory on turnera is CRITICAL: CRITICAL - 3.4% (282080 kB) free! [15:22:26] Platonides * Re: [Toolserver-l] How to have qsub mail output? [15:34:25] Tim Landscheidt * Re: [Toolserver-l] How to have qsub mail output? [15:35:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:43:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:44:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:45:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37745 MB (6% inode=99%): [15:47:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:52:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 492856 MB (9% inode=42%): [15:54:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:55:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:56:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:57:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:57:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:03:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106925 MB (10% inode=99%): [16:08:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:09:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107221 MB (11% inode=99%): [16:10:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106798 MB (10% inode=99%): [16:14:59] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:20:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (371816 kB) free! [16:29:59] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:33:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107239 MB (11% inode=99%): [16:35:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:43:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:44:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:45:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37500 MB (6% inode=99%): [16:47:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:54:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 492657 MB (9% inode=42%): [16:54:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:55:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:57:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:57:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:58:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:03:26] Carl (CBM) * Re: [Toolserver-l] Future of the toolserver [17:08:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:17:21] apmon, DaBPunkt needs to get confirmation from an OSM project member that I can be added to the project , too. [17:17:48] there is no urgency now, as i'm leaving for vacation on Wednesday anyways [17:21:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (400012 kB) free! [17:29:58] Free Memory on turnera is WARNING: WARNING - 5.1% (428584 kB) free! [17:35:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:38:41] hey, any sysadmins free to talk about the toolserver IP allocation ? [17:38:50] mark tried emailing, but no luck [17:39:01] * Damianz pokes DaBPunkt or nosy1 [17:39:29] @replag [17:39:31] jem-: s3-rr-a: 12s [-0.00 s/s]; s3-user: 12s [-0.00 s/s] [17:39:37] :) [17:39:50] I really gotta fix my client so I can be in here more often [17:39:55] i mean, we can talk about it by shutting off reachability, but making proper ip requests would be much much better [17:40:02] this stupid flood on autojoin is the worst ever [17:41:02] LeslieCarr: We got the mail [17:41:09] yay DaBPunkt [17:41:11] glad you got it [17:41:27] mark and i would be more than happy to help you with filling out the paperworks [17:41:53] But I have to say that I'm confused. Whos idea was it to requests new IPs now that RIPE is down to its last /8-block [17:42:13] heh, yeah we were both at RIPE last week [17:42:31] so RIPE is getting stricter about us actually properly using our IP space [17:42:38] right now toolserver is under WMF's PI space [17:42:48] and PI space is not by policy reallocatable [17:42:57] we should be properly doing this by being a LIR and allocating you PA space [17:43:20] what is a LIR? [17:43:35] local internet registry [17:43:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:44:06] um, in real english it's someone who has a voting stake in RIPE and can allocate ip space for themselves and small chunks for others [17:44:31] so the wmf becomes a LIR? [17:44:52] yep [17:45:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37258 MB (6% inode=99%): [17:45:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:47:11] and we'd allocate toolserver a /27 or /26 (if a /26 is needed) [17:47:11] LeslieCarr: I understood taht we (the TS) has to request a IP-rang (throu you). What I not understand: Do we have to build something like a LIR too for this? [17:47:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:47:19] you do not have to build an LIR [17:47:49] you'd be requesting space from a sponsoring LIR (i know, so much RIPE-specific speak) [17:48:04] (I have a RIPE-id for myself, but AFAIK the TS has not) [17:48:12] cool [17:49:38] LeslieCarr: I should have time somewhen next week (or maybe this week-end). Can I send you or mark questions if something is unclear for me? [17:49:54] sounds good [17:49:59] send to both i'd say [17:50:55] I'm sorry that we not replied yet BTW. [17:51:13] no problem, i'm assuming it's because you're busy ? [17:51:25] yes [17:51:35] I guess a /27 is more than enough BTW [17:51:45] busy is a totally valid reason - you guys have your /27 now which seems to be good [17:52:57] the routing will still be done by WMF? Because we have no router in our rack [17:54:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:54:16] yep [17:54:19] we will still do all the routing [17:54:37] ok [17:54:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 492311 MB (9% inode=42%): [17:55:49] DaBPunkt: dschwen: Is there a jira ticket where I can approve dschwen to the OSM project? Or can I do it here over IRC? [17:55:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:56:22] apmon: https://jira.toolserver.org/browse/TS-1525 [17:57:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:57:31] DaBPunkt: i'm going to be signing off this channel , do you have my email ? [17:57:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:57:38] LeslieCarr: yes [17:57:41] cool [17:57:43] good bye [17:57:44] ttfn [17:58:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:00:41] *** Reboot of the linux-boxes will start in 1h *** [18:00:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (396828 kB) free! [18:02:50] DaBPunkt: Thanks, I have replied to the jira ticket. [18:04:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106907 MB (10% inode=99%): [18:05:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106995 MB (11% inode=99%): [18:06:18] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 8749 MB (0% inode=93%): [18:08:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:12:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106892 MB (10% inode=99%): [18:30:58] Free Memory on turnera is WARNING: WARNING - 5.1% (427592 kB) free! [18:35:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:37:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.1% (341460 kB) free! [18:43:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:45:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37041 MB (6% inode=99%): [18:45:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:47:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:51:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106895 MB (10% inode=99%): [18:54:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:54:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 491551 MB (9% inode=42%): [18:55:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:57:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:57:39] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:58:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:05:00] if you have still open files on yarrow: close them NOW! [19:08:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:14:43] yarrow is done, next nightshade [19:14:59] Free Memory on turnera is WARNING: WARNING - 5.1% (430764 kB) free! [19:17:38] NTP on yarrow is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.02729 secs [19:20:30] nightshade is also done [19:25:16] mayapple also. Rebooting finish [19:26:37] NTP on yarrow is OK: NTP OK: Offset -0.070477 secs [19:26:38] NTP on nightshade is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset 0.004177 secs [19:26:38] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107710 MB (11% inode=99%): [19:28:26] DaB. * Re: [Toolserver-announce] Reboot of the linux-boxes at Monday 19:05 UTC [19:29:47] NTP on mayapple is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset 0.009802 secs [19:35:39] NTP on nightshade is OK: NTP OK: Offset 0.005694 secs [19:36:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:38:47] NTP on mayapple is OK: NTP OK: Offset 0.01961 secs [19:43:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:45:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36755 MB (6% inode=99%): [19:45:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:46:00] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (309644 kB) free! [19:47:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:54:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:54:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 493939 MB (9% inode=42%): [19:55:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:56:54] hello [19:57:06] will stop s2 and s5 soon [19:57:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:57:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:01] hello nosy1 :-) [19:58:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:58:22] hello DaBPunkt :) [20:04:36] I will now stop nfs on hemlock to look if that breaks the webservers [20:05:38] MySQL slave on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [20:05:59] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [20:05:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [20:05:59] MySQL on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [20:06:21] ok, it breaks it [20:06:51] shutdown of mysql completed as you see, move of the data started [20:09:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:14:58] Free Memory on turnera is WARNING: WARNING - 5.1% (427956 kB) free! [20:15:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417304 kB) free! [20:17:59] Free Memory on turnera is WARNING: WARNING - 5.2% (436664 kB) free! [20:30:38] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1891 [20:30:59] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1894.000000 [20:30:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1960.000000 [20:30:59] MySQL on cassia is OK: Uptime: 36 Threads: 6 Questions: 9650 Slow queries: 1 Opens: 192 Flush tables: 1 Open tables: 185 Queries per second avg: 268.55 [20:31:38] MySQL slave on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [20:31:59] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [20:31:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [20:32:18] /sql on cassia is OK: DISK OK - free space: /sql 227196 MB (19% inode=99%): [20:32:39] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1911 [20:32:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2079.000000 [20:32:59] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1927.000000 [20:34:51] ok, maintenance done - everything running again :) [20:35:39] gratulation :) [20:36:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:38:25] Marlen Caemmerer * Re: [Toolserver-l] Maintenance of s2 and s5 Monday, 1st October [20:39:18] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.023 second response time [20:41:08] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.104 second response time [20:41:47] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 8.176 second response time [20:42:37] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.019 second response time [20:43:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:45:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36585 MB (5% inode=99%): [20:45:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:48:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:54:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:54:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 493463 MB (9% inode=42%): [20:55:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:56:25] DaB. * Re: [Toolserver-announce] Maintenance-window for webservers Monday 20:00-22:00 UTC [20:57:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:58:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:58:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:01:29] gn8 [21:08:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (379524 kB) free! [21:09:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:10:39] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39763 MB (9% inode=99%): [21:10:59] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 78780 MB (8% inode=98%): [21:11:09] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 78780 MB (8% inode=98%): [21:36:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:45:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36443 MB (5% inode=99%): [21:46:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:48:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:54:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:54:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 493347 MB (9% inode=42%): [21:55:58] Free Memory on turnera is WARNING: WARNING - 5.1% (424152 kB) free! [21:56:59] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422072 kB) free! [21:56:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:57:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:58:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:58:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:59] Free Memory on turnera is WARNING: WARNING - 5.2% (433924 kB) free! [22:09:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:19:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (403396 kB) free! [22:33:10] nacht ts [22:36:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:44:58] Free Memory on turnera is WARNING: WARNING - 5.4% (454032 kB) free! [22:45:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36340 MB (5% inode=99%): [22:46:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:48:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:54:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:54:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 491517 MB (9% inode=42%): [22:56:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:57:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:58:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:58:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:03:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (375148 kB) free! [23:09:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:20:58] Free Memory on turnera is OK: OK - 8.8% (733300 kB) free. [23:36:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:41:58] Free Memory on turnera is WARNING: WARNING - 5.7% (476960 kB) free! [23:44:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:45:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36110 MB (5% inode=99%): [23:46:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:48:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:53:59] Free Memory on turnera is OK: OK - 7.4% (617588 kB) free. [23:54:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:54:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 485814 MB (9% inode=42%): [23:56:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:57:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:58:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:58:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.