[00:02:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37557 MB (6% inode=99%): [00:08:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:16:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [00:16:51] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [00:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:20:51] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [00:21:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:29:13] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571465 MB (10% inode=46%): [00:30:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:53:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:57:12] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 116311 MB (11% inode=99%): [01:01:12] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 26399 MB (6% inode=99%): [01:02:12] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23384 MB (5% inode=99%): [01:02:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37538 MB (6% inode=99%): [01:08:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:16:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [01:16:51] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [01:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:20:52] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [01:21:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:30:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571364 MB (10% inode=46%): [01:30:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:53:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:02:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37545 MB (6% inode=99%): [02:08:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:17:50] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [02:17:50] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [02:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:21:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:21:52] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [02:30:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 570050 MB (10% inode=46%): [02:31:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:53:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37495 MB (6% inode=99%): [03:08:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:15:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:17:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [03:17:51] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [03:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:21:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:21:52] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [03:30:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 569771 MB (10% inode=46%): [03:31:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:53:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:03:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37480 MB (6% inode=99%): [04:09:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:17:52] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [04:17:52] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [04:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:22:52] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [04:30:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 569193 MB (10% inode=46%): [04:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:53:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:04:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37465 MB (6% inode=99%): [05:09:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:15:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:17:52] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [05:18:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [05:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:23:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [05:30:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 568045 MB (10% inode=46%): [05:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:53:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:12] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 16875 MB (1% inode=96%): [06:04:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37444 MB (6% inode=99%): [06:09:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:18:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [06:18:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [06:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:23:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [06:30:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 568678 MB (10% inode=46%): [06:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:54:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:05:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37420 MB (6% inode=99%): [07:09:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:15:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:18:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [07:18:50] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [07:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:23:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [07:31:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 568572 MB (10% inode=46%): [07:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:55:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:04:12] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29094 MB (7% inode=99%): [08:05:12] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23080 MB (5% inode=99%): [08:06:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37393 MB (6% inode=99%): [08:09:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:16:13] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 86474 MB (8% inode=98%): [08:18:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [08:18:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [08:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:23:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [08:31:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 568332 MB (10% inode=46%): [08:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:55:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:58:02] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2000.000000 [08:58:02] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2002 [09:06:20] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37299 MB (6% inode=99%): [09:09:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:18:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [09:19:01] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1740.000000 [09:19:01] MySQL slave on rosemary is OK: Uptime: 13543300 Threads: 11 Questions: 5822641335 Slow queries: 1710133 Opens: 194718 Flush tables: 6 Open tables: 3839 Queries per second avg: 429.927 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1733 [09:19:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [09:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:23:02] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [09:31:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 568127 MB (10% inode=46%): [09:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:55:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:06:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37203 MB (6% inode=99%): [10:07:11] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 43734 MB (10% inode=99%): [10:09:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:15:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:18:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [10:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:20:50] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [10:22:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:24:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [10:31:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 567849 MB (10% inode=46%): [10:31:11] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 86181 MB (8% inode=98%): [10:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:56:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:07:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 37055 MB (6% inode=99%): [11:09:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:18:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [11:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:20:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [11:23:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:25:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [11:31:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 567596 MB (10% inode=46%): [11:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:45:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:56:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:07:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36986 MB (6% inode=99%): [12:09:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:18:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [12:20:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:20:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [12:23:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:25:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [12:31:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 567406 MB (10% inode=46%): [12:32:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:54:12] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 24767 MB (6% inode=99%): [12:55:11] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22717 MB (5% inode=99%): [12:56:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:57:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 116602 MB (12% inode=99%): [13:07:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36832 MB (6% inode=99%): [13:09:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:19:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [13:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:21:52] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [13:23:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:25:02] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [13:31:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 567213 MB (10% inode=46%): [13:33:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:57:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:01:44] @replag [14:01:44] DarkoNeko: s2-user-c: 4m 23s [-0.01 s/s]; s5-user-c: 4m 23s [-0.01 s/s] [14:06:32] the way the cron manager is totally bottlenecked is worrying :( [14:07:41] I see there has been profiling about how many task there were per minute, but have the responsible users (if any) be pinpointed too ? [14:08:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36787 MB (6% inode=99%): [14:08:38] (I mean, not to show them publicly, but telling them in private aand/or cutting the responsible parts, etc etc) [14:09:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:11:50] hello all [14:15:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:19:01] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [14:20:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:22:52] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [14:23:06] hello DaBPunkt [14:23:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:25:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [14:31:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 566962 MB (10% inode=46%): [14:33:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:57:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:09:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36649 MB (5% inode=99%): [15:10:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:16:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:19:00] APT on mayapple is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [15:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:23:51] APT on nightshade is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [15:24:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:25:01] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [15:31:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 566676 MB (10% inode=46%): [15:33:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:38:01] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:38:51] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:40:00] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:46:01] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2053.000000 [15:57:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:09:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36615 MB (5% inode=99%): [16:10:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:16:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:24:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:32:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 566437 MB (10% inode=46%): [16:33:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:40:01] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [16:45:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:57:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:09:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36541 MB (5% inode=99%): [17:10:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:13:40] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2090 [17:16:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:18:31] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1972 [17:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:23:01] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3561.000000 [17:24:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:30:21] / on willow is WARNING: DISK WARNING - free space: / 21976 MB (20% inode=99%): [17:30:22] /tmp on willow is WARNING: DISK WARNING - free space: / 21976 MB (20% inode=99%): [17:31:01] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1688.000000 [17:32:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 566260 MB (10% inode=46%): [17:34:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:45:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:55:12] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 18525 MB (1% inode=96%): [17:57:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:10:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:10:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36442 MB (5% inode=99%): [18:13:41] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2012 [18:16:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:19:31] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2536 [18:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:20:41] MySQL slave on z-dat-s3-a is OK: Uptime: 2616724 Threads: 18 Questions: 2170969422 Slow queries: 142459 Opens: 31020466 Flush tables: 1 Open tables: 16384 Queries per second avg: 829.651 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1751 [18:23:32] MySQL slave on z-dat-s7-a is OK: Uptime: 9901688 Threads: 13 Questions: 2588323268 Slow queries: 336533 Opens: 22173073 Flush tables: 1 Open tables: 7235 Queries per second avg: 261.402 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1751 [18:24:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:30:21] / on willow is WARNING: DISK WARNING - free space: / 21880 MB (20% inode=99%): [18:31:21] /tmp on willow is WARNING: DISK WARNING - free space: / 21877 MB (20% inode=99%): [18:32:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 565990 MB (10% inode=46%): [18:34:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:46:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:57:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:10:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:10:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36324 MB (5% inode=99%): [19:16:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:24:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:30:21] / on willow is WARNING: DISK WARNING - free space: / 21771 MB (20% inode=99%): [19:31:21] /tmp on willow is WARNING: DISK WARNING - free space: / 21770 MB (20% inode=99%): [19:33:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 565731 MB (10% inode=46%): [19:34:21] /tmp on willow is OK: DISK OK - free space: / 24308 MB (23% inode=99%): [19:34:21] / on willow is OK: DISK OK - free space: / 24308 MB (23% inode=99%): [19:34:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:46:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:57:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:02:37] DaBPunkt, can you take a look at ACCAPP-565 ? [20:03:52] Platonides: I plan to do later today [20:04:12] ok [20:10:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:10:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36210 MB (5% inode=99%): [20:16:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:17:12] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 84436 MB (8% inode=98%): [20:20:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:24:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:33:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 565419 MB (10% inode=46%): [20:34:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:46:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:58:01] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:10:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:10:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36071 MB (5% inode=99%): [21:16:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:25:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:33:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 565130 MB (10% inode=46%): [21:34:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:46:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:58:00] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:07:21] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 43084 MB (10% inode=99%): [22:10:21] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:10:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35970 MB (5% inode=99%): [22:16:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:25:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:31:21] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 83737 MB (8% inode=98%): [22:33:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 564985 MB (10% inode=46%): [22:34:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:46:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:48:43] nacht ts [22:56:21] Load avg. on adenia is WARNING: WARNING - load average: 15.16, 13.77, 8.48 [22:58:01] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:58:21] Load avg. on adenia is OK: OK - load average: 15.00, 14.17, 9.29 [23:10:22] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:10:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35883 MB (5% inode=99%): [23:16:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:20:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:25:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:33:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 563632 MB (10% inode=46%): [23:34:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:37:21] Load avg. on adenia is WARNING: WARNING - load average: 15.04, 15.14, 14.62 [23:46:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:48:22] Load avg. on adenia is OK: OK - load average: 11.38, 14.15, 14.55 [23:58:00] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.