[00:05:05] MySQL slave on rosemary is OK: Uptime: 14633260 Threads: 11 Questions: 6443885745 Slow queries: 1979657 Opens: 264950 Flush tables: 6 Open tables: 3851 Queries per second avg: 440.358 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1792 [00:05:34] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1723.000000 [00:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:29:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:31:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51043 MB (8% inode=99%): [00:37:26] Free Memory on turnera is WARNING: WARNING - 5.1% (424784 kB) free! [00:37:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 518252 MB (9% inode=44%): [00:37:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:37:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:38:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:39:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 294574 [00:40:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:40:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:43:26] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (419860 kB) free! [00:43:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 281845 [00:48:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:52:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 348357 [00:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:55:26] Free Memory on turnera is WARNING: WARNING - 5.1% (429992 kB) free! [00:58:26] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (414864 kB) free! [01:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:29:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:31:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50951 MB (8% inode=99%): [01:36:26] Free Memory on turnera is WARNING: WARNING - 5.3% (446052 kB) free! [01:37:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:38:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 518228 MB (9% inode=44%): [01:38:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 270962 [01:40:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:40:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:43:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (405844 kB) free! [01:43:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 253691 [01:48:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:52:26] Free Memory on turnera is WARNING: WARNING - 5.1% (431128 kB) free! [01:52:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 321113 [01:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:01:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (382120 kB) free! [02:03:26] Free Memory on turnera is WARNING: WARNING - 5.9% (491460 kB) free! [02:06:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106896 MB (10% inode=99%): [02:13:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107270 MB (11% inode=99%): [02:14:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106777 MB (10% inode=99%): [02:16:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (387864 kB) free! [02:28:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:29:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:31:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50691 MB (8% inode=99%): [02:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:38:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:39:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 516195 MB (9% inode=44%): [02:39:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 227057 [02:40:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:40:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:44:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 210876 [02:49:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:52:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 301012 [02:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:19:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:28:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:29:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:29:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109349 MB (11% inode=99%): [03:31:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50447 MB (8% inode=99%): [03:38:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:38:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:39:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515196 MB (9% inode=43%): [03:39:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:39:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:39:45] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 202966 [03:40:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:41:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:45:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180907 [03:49:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:51:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (364260 kB) free! [03:52:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 283792 [03:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:54:26] Free Memory on turnera is WARNING: WARNING - 6.6% (549668 kB) free! [04:01:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (410464 kB) free! [04:06:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106763 MB (10% inode=99%): [04:07:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107175 MB (11% inode=99%): [04:17:26] Free Memory on turnera is WARNING: WARNING - 6.1% (514324 kB) free! [04:19:26] Free Memory on turnera is OK: OK - 8.1% (676312 kB) free. [04:29:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:29:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:31:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50375 MB (8% inode=99%): [04:38:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:38:26] Free Memory on turnera is WARNING: WARNING - 6.3% (525568 kB) free! [04:39:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515920 MB (9% inode=44%): [04:39:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 154546 [04:40:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:41:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:41:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:45:55] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 137542 [04:50:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:52:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 251545 [04:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:12:26] Free Memory on turnera is CRITICAL: CRITICAL - 1.8% (149324 kB) free! [05:29:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:29:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:32:27] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50302 MB (8% inode=99%): [05:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:39:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515853 MB (9% inode=44%): [05:39:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 143176 [05:40:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:41:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:41:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:46:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 117411 [05:50:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:52:47] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 233699 [05:52:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:01:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106826 MB (10% inode=99%): [06:02:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107400 MB (11% inode=99%): [06:05:26] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 6754 MB (0% inode=92%): [06:13:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.5% (213608 kB) free! [06:30:25] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:30:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:31:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105624 MB (10% inode=99%): [06:32:27] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50197 MB (8% inode=99%): [06:32:27] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108626 MB (11% inode=99%): [06:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:39:27] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515660 MB (9% inode=44%): [06:39:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:39:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123035 [06:40:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:41:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:42:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:46:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78668 [06:50:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:51:46] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36815 MB (9% inode=99%): [06:52:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 216157 [06:53:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:06:06] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 80417 MB (8% inode=98%): [07:06:15] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 80413 MB (8% inode=98%): [07:13:25] Free Memory on turnera is CRITICAL: CRITICAL - 2.9% (245020 kB) free! [07:30:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:30:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50076 MB (8% inode=99%): [07:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:39:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515512 MB (9% inode=44%): [07:39:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:39:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 83165 [07:40:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:41:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:42:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:47:56] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40413 [07:50:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:52:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 188412 [07:53:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:13:25] Free Memory on turnera is CRITICAL: CRITICAL - 3.1% (261336 kB) free! [08:30:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:30:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49935 MB (8% inode=99%): [08:37:55] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3092 [08:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:39:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:39:56] MySQL slave on z-dat-s7-a is OK: Uptime: 304449 Threads: 9 Questions: 44531909 Slow queries: 78223 Opens: 68223 Flush tables: 1 Open tables: 5653 Queries per second avg: 146.270 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1744 [08:40:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63794 [08:40:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515111 MB (9% inode=43%): [08:40:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:40:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:41:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:42:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:44:35] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3274 MB (10% inode=93%): [08:44:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3274 MB (10% inode=93%): [08:51:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:52:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 161321 [08:53:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:13:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.6% (220688 kB) free! [09:21:36] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3297 MB (11% inode=93%): [09:21:36] / on wolfsbane is WARNING: DISK WARNING - free space: / 3297 MB (11% inode=93%): [09:22:35] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3279 MB (10% inode=93%): [09:22:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3279 MB (10% inode=93%): [09:25:35] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3298 MB (11% inode=93%): [09:25:35] / on wolfsbane is WARNING: DISK WARNING - free space: / 3298 MB (11% inode=93%): [09:30:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:30:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49747 MB (8% inode=99%): [09:38:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:40:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30779 [09:40:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 514897 MB (9% inode=43%): [09:40:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:40:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:40:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:42:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:42:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:45:34] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3223 MB (10% inode=93%): [09:45:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3223 MB (10% inode=93%): [09:51:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:52:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139001 [09:53:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:01:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106827 MB (10% inode=99%): [10:02:25] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107364 MB (11% inode=99%): [10:14:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (251480 kB) free! [10:15:26] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3367 [10:20:26] MySQL slave on z-dat-s6-a is OK: Uptime: 45834 Threads: 8 Questions: 15822091 Slow queries: 2238 Opens: 1994 Flush tables: 1 Open tables: 1470 Queries per second avg: 345.204 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1433 [10:24:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106798 MB (10% inode=99%): [10:25:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107184 MB (11% inode=99%): [10:30:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:31:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:32:27] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49509 MB (8% inode=99%): [10:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:40:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 514710 MB (9% inode=43%): [10:40:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:40:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:41:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:42:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:42:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:45:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 495 MB (1% inode=93%): [10:45:35] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 495 MB (1% inode=93%): [10:51:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107124 [10:53:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:15:25] Free Memory on turnera is CRITICAL: CRITICAL - 2.3% (194232 kB) free! [11:21:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106568 MB (10% inode=99%): [11:21:32] @replag [11:21:36] Hydriz: s1-rr-a: 30s [+0.00 s/s]; s1-user: 31s [+0.00 s/s]; s2-user: 36m 12s [+0.04 s/s]; s2-user-c: 6m 15s [+0.01 s/s]; s3-rr-a: 1d 1h 18m 37s [-5.35 s/s]; s3-user: 1d 1h 18m 37s [-5.35 s/s]; s5-user-c: 6m 15s [+0.01 s/s] [11:21:53] hmmph, one day for s3 [11:25:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108359 MB (11% inode=99%): [11:30:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:31:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49283 MB (8% inode=99%): [11:38:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:40:27] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 514174 MB (9% inode=43%): [11:40:27] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:40:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:41:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:42:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:42:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:45:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 468 MB (1% inode=93%): [11:45:35] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 468 MB (1% inode=93%): [11:52:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:52:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 76106 [11:53:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:13:08] wow, s6 works, happy me) [12:15:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (233360 kB) free! [12:16:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106905 MB (10% inode=99%): [12:21:26] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1948.000000 [12:30:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:31:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49053 MB (8% inode=99%): [12:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:40:25] Free Memory on turnera is WARNING: WARNING - 5.1% (426152 kB) free! [12:40:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 513979 MB (9% inode=43%): [12:40:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:40:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:41:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:42:25] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (362320 kB) free! [12:42:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:42:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:45:34] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 410 MB (1% inode=93%): [12:45:34] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 410 MB (1% inode=93%): [12:52:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:52:45] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56497 [12:53:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:19:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [13:19:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:24:26] Free Memory on turnera is WARNING: WARNING - 5.1% (426420 kB) free! [13:30:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (313848 kB) free! [13:30:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:31:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48820 MB (7% inode=99%): [13:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:40:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:41:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 513763 MB (9% inode=43%): [13:41:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:41:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:42:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:42:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:44:46] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:45:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 311 MB (1% inode=93%): [13:45:35] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 311 MB (1% inode=93%): [13:52:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:52:46] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13668 [13:53:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:15:45] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3449 [14:19:27] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4842.000000 [14:20:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 109309 MB (11% inode=99%): [14:20:45] MySQL slave on z-dat-s3-a is OK: Uptime: 156519 Threads: 15 Questions: 99389624 Slow queries: 118453 Opens: 1127263 Flush tables: 1 Open tables: 16384 Queries per second avg: 635.0 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1704 [14:30:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.9% (245188 kB) free! [14:30:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:31:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48610 MB (7% inode=99%): [14:38:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:40:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:41:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 513586 MB (9% inode=43%): [14:41:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:41:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:42:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:42:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:44:12] @replag [14:44:13] Hydriz: s2-user: 2h 57m 55s [+0.70 s/s]; s2-user-c: 1h 17m 5s [+0.35 s/s]; s3-rr-a: 17s [-7.49 s/s]; s3-user: 17s [-7.49 s/s]; s5-user-c: 1h 17m 5s [+0.35 s/s] [14:45:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 254 MB (0% inode=93%): [14:45:35] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 254 MB (0% inode=93%): [14:50:46] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [14:53:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:54:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:20:27] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4258.000000 [15:30:25] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (236264 kB) free! [15:30:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:30:27] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106923 MB (10% inode=99%): [15:31:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48395 MB (7% inode=99%): [15:38:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:38:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107772 MB (11% inode=99%): [15:40:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:41:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:41:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 512838 MB (9% inode=43%): [15:41:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:42:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:43:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:45:35] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 179 MB (0% inode=93%): [15:45:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 179 MB (0% inode=93%): [15:50:55] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [15:53:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:54:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:20:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5707.000000 [16:30:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.3% (191436 kB) free! [16:30:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:31:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:32:25] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48151 MB (7% inode=99%): [16:34:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106924 MB (10% inode=99%): [16:38:26] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:39:11] I'm having a problem with a script running from qcronsub. It's a sh script, i've already made chmod a+x but when qcronsub runs it, i got the message that it's not an executable one [16:40:39] Merlissimo: ^ [16:41:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:41:45] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:42:25] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:42:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 512367 MB (9% inode=43%): [16:42:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:44:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:45:34] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 102 MB (0% inode=93%): [16:45:34] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 102 MB (0% inode=93%): [16:49:34] Alchimista: can you run it from the shell? [16:49:55] i.e. alchimista@server $ /path/to/script ? [16:51:46] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [16:53:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:55:03] valhallasw: yap [16:55:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108314 MB (11% inode=99%): [16:56:57] valhallasw: i can runnit form shell with sh aleph.sh, for example [16:57:46] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [16:57:52] valhallasw: in qcronsub i'm using: /usr/bin/sh /home/alchimista/aleph/avbot.sh . i've tryed also /bin/sh, but didn't worked [17:10:17] can't you use #! ? [17:12:07] giftpflanze: # -*- coding: utf-8 -*-? all the scripts has it [17:20:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6783.000000 [17:30:25] Free Memory on turnera is CRITICAL: CRITICAL - 2.7% (222836 kB) free! [17:31:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:31:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:32:26] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47974 MB (7% inode=99%): [17:38:29] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:41:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:41:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:42:26] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:42:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 512192 MB (9% inode=43%): [17:42:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:44:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:45:35] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 32 MB (0% inode=93%): [17:45:35] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 32 MB (0% inode=93%): [17:51:45] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [17:53:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:54:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:57:46] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [17:59:45] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [18:00:37] Alchimista: no, #!/usr/bin/bash [18:02:34] valhallasw: you mean on qcronsub or in the bash script? [18:02:54] in the script [18:03:02] and then call the script directly from qcronsub [18:03:47] well yes, i allways use it [18:05:27] there is no avbot.sh in that folder? [18:05:32] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 1444 MB (0% inode=92%): [18:08:51] if i use the path to the file wich is on mu crontab, it works, so seems to be fine [18:10:23] is it possible to set PYTHONIOENCODING from qcronsub? That's why i'm tryng to do with am sh script [18:20:27] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7546.000000 [18:30:54] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (174804 kB) free! [18:32:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:32:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:32:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47678 MB (7% inode=99%): [18:36:19] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106855 MB (10% inode=99%): [18:38:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:41:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107438 MB (11% inode=99%): [18:41:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:42:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:42:30] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:42:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 511526 MB (9% inode=43%): [18:43:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:45:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:46:42] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [18:46:42] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=93%): [18:52:15] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38307 MB (9% inode=99%): [18:53:07] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [18:53:07] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:53:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:53:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:55:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:58:26] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [18:59:04] /tmp on wolfsbane is OK: DISK OK - free space: / 17128 MB (57% inode=93%): [18:59:05] / on wolfsbane is OK: DISK OK - free space: / 17127 MB (57% inode=93%): [19:00:15] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [19:06:38] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 80240 MB (8% inode=98%): [19:06:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 80240 MB (8% inode=98%): [19:20:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8317.000000 [19:25:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106792 MB (10% inode=99%): [19:26:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107338 MB (11% inode=99%): [19:30:59] Free Memory on turnera is CRITICAL: CRITICAL - 3.3% (276216 kB) free! [19:32:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:32:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:32:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47362 MB (7% inode=99%): [19:39:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:42:25] Samuel Klein * Re: [Toolserver-l] Future of the toolserver [19:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:42:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:42:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 511270 MB (9% inode=43%): [19:43:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:46:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:53:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:53:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:53:58] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [19:55:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:58:29] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [20:00:29] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [20:20:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7795.000000 [20:30:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.5% (210340 kB) free! [20:32:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47025 MB (7% inode=99%): [20:33:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:33:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:58] Free Memory on turnera is WARNING: WARNING - 5.1% (423304 kB) free! [20:39:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:42:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:42:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:42:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 511133 MB (9% inode=43%): [20:44:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:44:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (413680 kB) free! [20:46:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:53:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:53:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:53:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [20:56:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:58:29] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [21:00:29] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [21:10:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106920 MB (10% inode=99%): [21:20:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6120.000000 [21:22:12] @replag [21:22:13] Jag: s1-rr-a: 18m 52s [+0.03 s/s]; s1-user: 18m 53s [+0.03 s/s]; s2-user: 4h 5m 48s [+0.17 s/s]; s2-user-c: 1h 41m 0s [+0.06 s/s]; s3-rr-a: 1m 58s [+0.00 s/s]; s3-user: 1m 58s [+0.00 s/s]; s5-user-c: 1h 41m 0s [+0.06 s/s] [21:24:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106967 MB (11% inode=99%): [21:25:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106770 MB (10% inode=99%): [21:30:53] great thanks to DaB or somebody, s6 is up) [21:32:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46857 MB (7% inode=99%): [21:33:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:33:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:39:48] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2095 [21:40:08] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2117.000000 [21:40:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:42:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:42:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 511029 MB (9% inode=43%): [21:43:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:44:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:44:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (418604 kB) free! [21:46:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:49:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106692 MB (10% inode=99%): [21:52:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107241 MB (11% inode=99%): [21:53:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:53:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:53:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [21:53:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106830 MB (10% inode=99%): [21:56:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:56:58] Free Memory on turnera is WARNING: WARNING - 5.1% (425744 kB) free! [21:58:29] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [21:58:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107253 MB (11% inode=99%): [21:58:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (407840 kB) free! [22:00:29] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [22:02:29] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2124.000000 [22:06:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [22:06:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3638 [22:18:59] Free Memory on turnera is WARNING: WARNING - 5.1% (424424 kB) free! [22:19:59] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (414944 kB) free! [22:21:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6251.000000 [22:32:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46422 MB (7% inode=99%): [22:33:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:33:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:35:01] @replag [22:35:02] Platonides: s1-rr-a: 1h 26m 14s [+0.92 s/s]; s1-user: 1h 26m 14s [+0.92 s/s]; s2-user: 3h 42m 5s [-0.28 s/s]; s2-user-c: 1h 44m 32s [+0.13 s/s]; s3-rr-a: 50s [+0.00 s/s]; s3-user: 50s [+0.00 s/s]; s5-user-c: 1h 44m 32s [+0.13 s/s] [22:36:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106713 MB (10% inode=99%): [22:38:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107162 MB (11% inode=99%): [22:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3603.000000 [22:39:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106709 MB (10% inode=99%): [22:40:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:42:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:42:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 510860 MB (9% inode=43%): [22:43:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:43:42] @replag [22:43:43] Platonides: s1-rr-a: 1h 34m 18s [+0.93 s/s]; s1-user: 1h 34m 18s [+0.93 s/s]; s2-user: 3h 33m 6s [-1.04 s/s]; s2-user-c: 1h 42m 31s [-0.23 s/s]; s3-rr-a: 46s [-0.01 s/s]; s3-user: 46s [-0.01 s/s]; s5-user-c: 1h 42m 31s [-0.23 s/s] [22:44:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:46:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:53:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:53:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [22:54:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:57:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:59:29] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [22:59:58] Free Memory on turnera is WARNING: WARNING - 5.1% (424364 kB) free! [23:00:30] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [23:00:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (392484 kB) free! [23:03:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107084 MB (11% inode=99%): [23:04:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106391 MB (10% inode=99%): [23:05:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107006 MB (11% inode=99%): [23:06:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6931 [23:07:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6955.000000 [23:09:58] Free Memory on turnera is WARNING: WARNING - 6.4% (537000 kB) free! [23:21:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4234.000000 [23:28:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3564.000000 [23:29:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (383384 kB) free! [23:33:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:33:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:33:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46282 MB (7% inode=99%): [23:33:59] Free Memory on turnera is WARNING: WARNING - 5.2% (431744 kB) free! [23:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6110.000000 [23:40:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:40:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106812 MB (10% inode=99%): [23:42:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:42:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509501 MB (9% inode=43%): [23:43:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:44:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (410204 kB) free! [23:45:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:46:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:53:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:53:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [23:54:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:58:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:59:30] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates).