[00:00:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:04:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:30:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:31:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:31:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:35:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:37:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:38:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28230 MB (4% inode=99%): [00:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:51:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 525741 MB (9% inode=44%): [00:58:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 114862 MB (11% inode=99%): [01:01:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:05:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:30:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:31:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:31:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:35:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:37:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:39:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26735 MB (4% inode=99%): [01:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:51:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 525618 MB (9% inode=44%): [02:02:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:05:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:30:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:31:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:32:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:36:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:37:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:39:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28167 MB (4% inode=99%): [02:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:51:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 523442 MB (9% inode=44%): [03:02:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:06:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:30:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:32:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:32:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:36:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:38:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:40:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28150 MB (4% inode=99%): [03:45:49] it's frequently seen these days: DB connection error: Lost connection to MySQL server at 'reading initial communication packet', system error: 111 (sql-s2-user) [03:45:57] what's wrong [03:46:07] With long queries? [03:46:11] Long-running, I mean. [03:46:14] Or any query? [03:46:21] Which server specifically? [03:47:49] Brooke: it [03:48:08] 's a fan of apostrophes? [03:48:12] Brooke: it's a long-running task [03:48:25] Did you get an e-mail about your query being killed? [03:48:26] but all queries are almost the same [03:48:29] no [03:48:54] I only use s2-user so I don't know the situation of other servers [03:49:53] What's the query? [03:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:52:16] Brooke: I can't find the exact query there [03:52:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 521383 MB (9% inode=44%): [03:55:55] Brooke: but that task really uses DB a lot [04:02:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:06:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:30:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:33:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:33:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:37:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:38:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:40:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28125 MB (4% inode=99%): [04:50:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:53:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 521258 MB (9% inode=44%): [05:02:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:07:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:30:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:33:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:33:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:38:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:41:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28084 MB (4% inode=99%): [05:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:53:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 521189 MB (9% inode=44%): [05:57:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 14266 MB (1% inode=95%): [06:02:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:07:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:30:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:34:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:38:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:41:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28026 MB (4% inode=99%): [06:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:54:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 521021 MB (9% inode=44%): [07:03:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:07:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:31:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:33:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:35:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:38:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:41:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27968 MB (4% inode=99%): [07:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:54:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 520893 MB (9% inode=44%): [08:03:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:07:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:31:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:34:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:35:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:35:32] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2096.000000 [08:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:39:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:41:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27900 MB (4% inode=99%): [08:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:54:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 520687 MB (9% inode=44%): [09:04:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:08:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:29:32] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1768.000000 [09:29:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 83177 MB (8% inode=98%): [09:30:41] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 83441 MB (8% inode=98%): [09:30:42] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 41585 MB (10% inode=99%): [09:31:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:34:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:36:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:39:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:41:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27803 MB (4% inode=99%): [09:50:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:55:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 520436 MB (9% inode=44%): [10:04:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:08:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:31:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:34:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:36:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:39:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:41:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27708 MB (4% inode=99%): [10:50:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:55:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 516575 MB (9% inode=44%): [11:04:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:08:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:04] DaB. * Re: [Toolserver-l] z-dat-s4-a (s4-user) is down (was: Re: Reboot [11:31:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:34:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:37:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:39:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:41:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27600 MB (4% inode=99%): [11:50:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:54:08] hello all [11:55:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 516285 MB (9% inode=44%): [12:04:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:08:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:32:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:35:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:37:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:39:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:41:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27504 MB (4% inode=99%): [12:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:56:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515936 MB (9% inode=44%): [12:59:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 115262 MB (11% inode=99%): [13:04:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:09:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:32:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:35:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:37:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:39:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:40:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:41:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27414 MB (4% inode=99%): [13:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:56:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515631 MB (9% inode=44%): [13:57:04] Hersfold Wikipedia * Re: [Toolserver-l] z-dat-s4-a (s4-user) is down (was: Re: Reboot [14:05:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:09:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:32:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:35:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:37:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:41:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27331 MB (4% inode=99%): [14:51:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:56:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 515264 MB (9% inode=44%): [15:05:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:09:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:27:04] Platonides * Re: [Toolserver-l] z-dat-s4-a (s4-user) is down (was: Re: Reboot [15:32:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:35:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:37:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:41:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27256 MB (4% inode=99%): [15:42:33] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1895.000000 [15:51:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:57:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 514569 MB (9% inode=44%): [16:05:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:10:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:28:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000 [16:32:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:35:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:37:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:41:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27176 MB (4% inode=99%): [16:44:22] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:51:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:57:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 514191 MB (9% inode=44%): [17:06:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:10:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:52] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:28:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4002.000000 [17:33:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:35:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:42:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27095 MB (4% inode=99%): [17:51:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:56:32] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3524.000000 [17:57:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 511054 MB (9% inode=43%): [17:58:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 13136 MB (1% inode=95%): [18:06:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:10:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:33:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [18:34:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:35:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:42:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27001 MB (4% inode=99%): [18:51:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:57:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 510648 MB (9% inode=43%): [18:59:42] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58329 MB (5% inode=97%): [18:59:52] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58290 MB (5% inode=97%): [19:01:21] RAID on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:04:52] RAID on rosemary is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:06:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:06:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 82574 MB (8% inode=98%): [19:06:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 82573 MB (8% inode=98%): [19:10:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:25:21] What's up with SSO ? [19:25:28] Jira and Wiki both deny my accounts existance [19:27:39] I just got into jira [19:32:04] Krinkle * Re: [Toolserver-l] Jira Auth broken [19:34:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4458.000000 [19:35:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:38:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:41:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:42:53] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26902 MB (4% inode=99%): [19:49:31] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3585.000000 [19:51:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:57:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509940 MB (9% inode=43%): [20:06:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:10:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:35:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:35:32] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1685.000000 [20:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:39:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:41:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:41:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:42:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26806 MB (4% inode=99%): [20:51:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:55:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117293 MB (19% inode=99%): [20:56:52] /sql on ptolemy is OK: DISK OK - free space: /sql 213304 MB (35% inode=99%): [20:58:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509635 MB (9% inode=43%): [21:06:04] Marlen Caemmerer * Re: [Toolserver-l] Jira Auth broken [21:07:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:10:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:30:42] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42011 MB (10% inode=99%): [21:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:36:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:39:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:41:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:41:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:51:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:58:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509462 MB (9% inode=43%): [22:07:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:10:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:36:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:40:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:41:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:42:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:52:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:55:05] nacht ts [22:59:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509321 MB (9% inode=43%): [23:08:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:10:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:36:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:40:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:43:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:53:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:59:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 507972 MB (9% inode=43%):